Image  Cover  Sheet 


DSIS  Use  only: 


CLASSIFICATION 

UNCLASSIFIED 


Deliver  to: 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

28  MAR  2000  2' REPORT  TYPE 

3.  DATES  COVERED 

00-00-2000  to  00-00-2000 

4.  TITLE  AND  SUBTITLE 

Literature  Survey  and  Situation  Analysis 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Bernard  Moulin  et  Associes  Inc, 91  rue  Lockwell  Quebec, Quebec  QC  GJR 
IV6,  Canada, 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

_ _ _  ABSTRACT 

18.  NUMBER  19a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  OS 

unclassified  unclassified  unclassified  Report  (SAR) 

101 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Literature  Survey  and  Situation  Analysis 


by 

Dr.  Bernard  Moulin,  Dr.  Mohamad  Allouche,  Dr.  Hengameh  Irandoust 


Bernard  Moulin  et  Associes  Inc. 

91  rue  Lockwell  Quebec 
Quibec  QC  G1R  1V6,  Canada 
Tel:  (418)  524  7891 


Contract  No.  W770 1-9-28 18/A 


On  behalf  of 

DEPARTMENT  OF  NATIONAL  DEFENCE 


DREV  Scientific  Authority 
Dr.  Driss  Kettani 
(418)  844  4000  (ext.  4396) 


28  March  2000 


©  HER  MAJECTY  THE  QUEEN  IN  RIGHT  OF  CANADA  (2000) 
as  represented  by  the  Minister  of  National  Defence 

©  S  A  MAJESTE  LA  REINE  EN  DROIT  DU  CANADA  (2000) 
representee  par  le  Ministre  de  la  Defense  Nationale  du  Canada 


I 


Executive  Summary 


This  report  presents  the  results  of  a  literature  survey  on  the  domain  of  data  fusion  and 
related  topics  in  order  to  get  a  better  understanding  of  the  process  on  situation  analysis. 

This  work  takes  place  in  the  context  of  a  long  range  project  that  aims  at  developing  a 
decision  support  system  (DSS)  for  Operation  Room  Officers  (OROs)  in  the  HALIFAX 
class  frigate.  Given  the  large  volume  of  incoming  data  from  various  sources,  the  time 
constraints  and  the  characteristics  of  the  activities  taking  place  in  the  Operation  Room, 
DREV  researchers  have  investigated  for  several  years  the  applicability  of  data  fusion 
techniques  for  the  development  of  a  DSS  for  OROs.  In  the  open  literature  the  JDL  model 
has  been  widely  used  to  structure  the  main  activities  taking  place  in  the  data  fusion 
domain.  For  several  years,  DREV  researchers  have  developed  systems  mainly  for 
multisource  data  fusion  (JDL  model's  levels  0  and  1).  When  working  on  situation  and 
impact  assessment  (levels  2  and  3),  they  faced  several  difficulties  and  discovered  that  one 
main  limitation  of  the  JDL  model  is  that  not  enough  attention  is  given  to  a  fundamental 
element  in  the  Command  and  Control  (C2)  process:  the  person  who  makes  decision. 

In  several  military  domains,  researchers  have  noticed  that  data  fusion  techniques  cannot 
provide  on  their  own  a  complete  and  satisfactory  solution  to  automate  the  complex 
activities  taking  place  in  C2  settings.  This  can  be  explained  by  the  fact  that  such  settings 
involve  ill-structured  problems,  uncertain  and  dynamic  environments,  conflicting, 
shifting  or  ill-defined  goals,  time  constraints,  high  stakes  and  pressure.  Given  the  current 
technology  and  software  development  know-how,  it  is  impossible  to  develop  a  data 
fusion  system  that  would  efficiently  integrate  all  the  functionalities  that  are  pictured  in 
the  JDL  idealized  model  without  considering  the  fundamental  contribution  of  human 
operators.  Hence,  a  data  fusion  system  should  be  thought  of  as  technological  means  to 
support  human  decisions. 

How  can  such  a  system  be  useful  to  support  OROs'  decision  making  process?  During  the 
past  years  the  DREV  team  has  emphasized  the  importance  of  considering  the  cognitive 
aspects  of  C2  decision  making  processes,  and  more  specifically  of  providing  means  to 
develop  and  maintain  operators'  situation  awareness.  From  the  operator's  point  of  view, 
situation  awareness  can  be  informally  defined  as  "knowing  what's  going  on  so  that  you 
can  figure  out  what  to  do".  In  the  context  of  the  project  for  developing  a  DSS  for  OROs 
in  the  HALIFAX  Class  Frigat,  a  Cognitive  Task  Analysis  of  the  ORO's  position 
(Matthews  et  al.  1999)  has  been  recently  completed  and  provides  relevant  observations 
related  to  the  ORO's  cognitive  needs:  1)  Gaining  and  maintaining  situation  awareness  is  a 
key  issue  for  OROs;  2)  OROs  employ  a  variety  of  mental  pictures  and  mental  models  to 
achieve  their  cognitive  goals;  3)  Mission  preparation  has  great  significance  for 
establishing  mission  related  mental  models;  4)  Updating  situation  awareness  when 
coming  on  watch  is  critical  for  updating  ORO's  mental  models. 

These  cognitive  aspects  of  the  ORO's  activities  cannot  be  accounted  for  in  the  JDL 
model.  However,  they  are  most  important  if  one  wants  to  develop  a  decision  support 
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system  that  will  be  useful  to  OROs.  Hence,  it  seems  appropriate  to  adopt  a  wider 
perspective  by  examining  how  a  decision  support  system  can  take  advantage  of  data 
provided  by  a  data  fusion  system  and  present  them  in  a  way  that  fits  OROs'  mental 
models.  This  cognitive  fit  should  enable  OROs  to  gain  and  maintain  situation  awareness. 
In  this  context  the  Scientific  Authority  of  this  contract  proposed  us  to  investigate  the 
process  of  "situation  analysis"  that  can  be  defined  as  a  process  that  leads  a  person  or  a 
group  of  persons  to  a  state  of  situation  awareness. 

This  document  is  composed  of  3  parts:  The  master  document  in  which  we  report  the 
results  of  the  study  on  situation  analysis  and  our  recommendations  for  future  works; 
Annex  1  which  presents  a  synthesis  of  works  done  on  Situation  and  Threat  Assessment  in 
the  Data  Fusion  domain;  Annex  2  which  presents  a  synthesis  on  the  subject  of  situation 
awareness.  Numerous  references  on  data  fusion,  situation  assessement  and  situation 
awareness  have  been  recorded  in  an  electronic  bibliographic  data  base  under  the  software 
Endnote™. 

Here  we  summarize  the  main  elements  of  the  master  document.  We  first  present  the  main 
elements  of  the  widely-used  JDL  model  for  data  fusion  and  discuss  its  limitations  when 
considering  the  cognitive  aspects  of  the  ORO's  activities.  We  then  examine  the  main 
characteristics  of  situation  awareness  which  plays  a  key  role  in  the  decision  making 
process.  A  recent  cognitive  task  analysis  for  the  ORO  position  has  been  carried  out 
(Matthews  et  al.  1999).  We  mention  its  main  conclusions  which  emphasize  the  key  role 
played  by  situation  awareness  and  mental  models  for  ORO's  cognitive  activities. 

In  this  context,  we  introduce  the  notion  of  situation  analysis  as  it  was  defined  by  the 
Scientific  Authority  of  the  present  contract. 

Then,  we  discuss  the  key  elements  that  influence  situation  analysis  in  the  context  of  the 
Operation  Room.  We  show  that  situation  analysis  is  a  broad  process  that  takes  place  at 
several  levels  at  once:  at  the  level  of  each  team  member  and  at  a  more  global  level  which 
corresponds  to  the  global  situation  understanding  of  the  team  taken  as  a  whole. 

We  also  propose  and  discuss  generic  architectures  for  a  situation  analysis  support  system 
that  could  be  used  to  support  the  work  of  the  ORO's  team:  operators  and  their  supervisors 
as  well  as  the  ORO  himself.  From  this  presentation  of  the  different  levels  of  situation 
analysis  taking  place  in  an  Operation  Room  we  conclude  that  the  JDL  Data  Fusion  model 
may  not  directly  apply  to  a  hierarchically  organized  team  and  should  be  revised 
considering  the  notion  of  situation  analysis. 

Finally,  we  present  our  recommendations  for  the  next  steps  to  be  performed  toward  the 
creation  of  a  situation  analysis  support  system.  More  specifically,  we  recommed  that  a 
knowledge  engineering  approach  be  coupled  with  the  cognitive  task  analysis  of  the  ORO 
and  the  key  members  of  his  team.  The  knowledge  engineering  phase  of  the  project  should 
enable  knowledge  engineers  to  obtain  formal  models  that  match  the  mental  models  and 
pictures  enabling  an  ORO  to  gain  and  maintain  situation  awareness.  Such  formal  models 
will  be  used  as  the  key  knowledge  and  data  structures  on  which  the  situation  analysis 
support  system  would  be  built. 
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Sommaire 


Ce  rapport  presente  les  resultats  d'une  revue  de  litterature  sur  le  domaine  de  la  fusion  de 
donnees  et  des  sujets  associes  afin  d'avoir  une  meilleure  comprehension  du  processus 
d'analyse  de  situation. 

Ce  travail  se  situe  dans  le  contexte  d'un  projet  h  long  terme  qui  vise  le  developpement 
d'un  systeme  d'aide  a  la  decision  (SAD)  pour  les  Officiers  de  la  Salle  d'operations 
(OROs)  dans  la  fregate  de  classe  HALIFAX.  Etant  donne  le  fort  volume  des  donnees 
foumies  par  diverses  sources  de  donnees,  les  contraintes  de  temps  et  les  caracteristiques 
des  activites  qui  prennent  place  dans  la  Salle  d’operations,  les  chercheurs  du  CRDV  ont 
explore  depuis  plusieurs  ann6es  !  applicability  des  techniques  de  fusion  de  donnees  pour 
le  developpement  d'un  SAD  pour  les  OROs.  Dans  la  litterature  generate  le  module  JDL 
est  tres  utilise  pour  structurer  les  principales  activites  accomplies  dans  le  domaine  de  la 
fusion  des  donnees.  Depuis  plusieurs  amtees,  les  chercheurs  du  CRDV  ont  developpe  des 
systemes  qui  font  la  fusion  de  domtees  multi-sources  (niveaux  0  et  1  du  modele  JDL). 
Quand  ils  ont  explore  les  niveaux  devaluation  de  situation  ("situation  assessment")  et 
devaluation  d'impact  ("impact  assessment")  (niveaux  2  et  3  du  modele  JDL),  ils  ont 
rencontte  plusieurs  difficultes  et  decouvert  qu'une  des  principales  limitations  du  modele 
JDL  est  de  ne  pas  accorder  une  importance  suffisante  a  un  element  fondamental  dans  un 
processus  de  commandement  et  controle  (C2):  la  personne  qui  prend  les  decisions. 

Dans  plusieurs  domaines  militaires  des  chercheurs  ont  remarque  que  les  techniques  de 
fusion  de  donnees  ne  fournissent  pas  de  solution  complete  et  satisfaisante  pour 
automatiser  les  activites  complexes  qui  sont  accomplies  dans  un  environnement  de  C2. 
Ceci  peut  etre  explique  par  le  fait  que  ces  environnements  sont  incertains  et  dynamiques, 
que  les  problemes  sont  mal  compris,  que  les  buts  sont  changeants,  mal-definis  et  souvent 
conflictuels,  qu'il  y  a  des  contraintes  de  temps  importantes  et  que  les  enjeux  et  la  pression 
sont  tres  importants.  Compte  tenu  du  developppement  actuel  de  la  technologie  et  du 
savoir-faire  en  develppement  de  systemes,  il  semble  impossible  de  developper  un 
systeme  de  fusion  de  donnees  qui  integrerait  efficacement  les  fonctionnalites  proposees 
dans  le  modele  JDL  sans  considerer  la  contribution  fondamentale  des  operateurs 
humains.  Aussi,  un  systeme  de  fusion  de  donnees  devrait-il  etre  vu  comme  un  moyen 
technologique  de  supporter  les  decisions  humaines. 

Comment  un  tel  systeme  peut  etre  utile  au  processus  de  prise  de  decision  des  OROs?  Au 
cours  des  demieres  annees,  l'dquipe  du  CRDV  a  souligne  l'importance  de  considerer  les 
aspects  cognitifs  des  processus  de  decision  en  C2  et  plus  specifiquement  de  foumir  des 
moyens  de  developper  et  entretenir  la  conscience  de  situation  ("situation  awareness").  Du 
point  de  vue  de  l'operateur,  la  conscience  de  situation  peut  etre  informellement  definie 
comme  "connaftre  ce  qui  se  passe  de  fa?on  &  savoir  quoi  faire".  Dans  le  contexte  du 
projet  de  developpement  d'un  SAD  pour  les  OROs  des  fregates  de  classe  HALIFAX,  une 
analyse  cognitive  de  taches  du  poste  d'ORO  (Matthews  et  al.  1999)  a  ete  realisde 
recemment  et  foumit  plusieurs  observations  tres  pertinentes  au  sujet  des  besoins  cognitifs 
des  OROs:  1)  Former  et  entretenir  la  conscience  de  situation  est  un  facteur  cle  pour  les 
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OROs;  2)  les  OROs  utilisent  un  ensemble  d'images  et  de  modeles  mentaux  pour  realiser 
leurs  buts  cognitifs;  3)  La  preparation  de  mission  a  une  grande  importance  pour 
l'etablissement  des  modeles  mentaux;  4)  mettre  a  jour  la  conscience  de  situation  au 
moment  de  la  prise  de  quart  est  critique  pour  la  mise  &  jour  des  modeles  mentaux  de 
TORO. 

Le  modele  JDL  ne  peut  pas  rendre  compte  de  ces  aspects  cognitifs  des  activitds  de 
l'ORO.  Cependant,  ils  sont  tres  importants  si  on  veut  developper  un  SAD  qui  sera  utile 
aux  OROs.  Aussi,  il  semble  approprie  d' adopter  une  perspective  plus  large  et  d'examiner 
comment  un  SAD  peut  tirer  parti  des  donn6es  foumies  par  un  systeme  de  fusion  de 
donnees  et  les  presenter  d'une  fagon  qui  s'harmonise  avec  les  modeles  mentaux  des 
OROs.  Cette  harmonisation  devrait  permettre  aux  OROs  de  former  et  entretenir  une 
conscience  de  situation.  Dans  ce  contexte,  l'autoritd  scientifique  de  ce  contrat  nous  a 
propose  d'explorer  le  processus  d'analyse  de  situation  qui  peut  etre  defini  comme  un 
processus  qui  permet  a  une  personne  ou  k  un  groupe  de  personnes  d'atteindre  un  etat  de 
conscience  de  situation. 

Ce  document  est  compose  de  trois  parties:  le  Document  maitre  dans  lequel  nous 
rapportons  les  resultats  de  l'^tude  sur  l'analyse  de  situation  et  presentons  nos 
recommandations  pour  des  travaux  futurs;  1' Annexe  1  qui  presente  une  synthese  de 
travaux  realises  dans  les  domaines  de  1'evaluation  de  situation  et  1'evaluation  de  menace 
dans  un  contexte  de  fusion  de  donnees;  l'Annexe  2  qui  prdsente  une  synthese  de  travaux 
sur  le  sujet  de  la  conscience  de  situation.  De  nombreuses  references  sur  la  fusion  de 
donnees,  1'evaluation  de  situation  et  la  conscience  de  situation  ont  ete  repertoriees  dans 
une  base  de  donnees  bibliographiques  sous  le  logiciel  Endnote™. 

Ici  nous  resumons  les  principaux  Elements  du  document  maitre.  Nous  presentons  tout 
d'abord  les  principaux  elements  du  module  JDL  pour  la  fusion  de  donnees  et  discutons 
ses  limitations  relativement  aux  aspects  cognitifs  des  activites  des  OROs.  Ensuite  nous 
examinons  les  principales  caract6ristiques  de  la  conscience  de  situation  qui  joue  un  role 
fondamental  dans  le  processus  de  prise  de  decision.  Une  recente  analyse  cognitive  de 
taches  du  poste  d'ORO  a  ete  realisee  (Matthews  et  al.  1999).  Nous  mentionnons  ses 
principales  conclusions  qui  soulignent  le  r61e  fondamental  joue  par  la  conscience  de 
situation  et  les  modules  mentaux  dans  les  activity  cognitives  des  OROs.  Dans  ce 
contexte  nous  introduisons  la  notion  d'analyse  de  situation  qui  a  6t6  definie  par  l'autorite 
scientifique  du  present  contrat. 

Par  la  suite  nous  discutons  des  elements  fondamentaux  qui  influencent  l'analyse  de 
situation  dans  le  contexte  de  la  Salle  d'opdrations.  Nous  montrons  que  l'analyse  de 
situation  est  un  large  processus  qui  se  deroule  simultannement  k  plusieurs  niveaux:  au 
niveau  de  chaque  membre  de  l'equipe  de  l'ORO  ainsi  qu'&  un  niveau  plus  global  qui 
correspond  a  la  comprehension  globale  de  la  situation  qui  caracterise  l'equipe  dans  son 
ensemble.  Nous  proposons  et  commentons  des  architectures  generiques  pour  un  systeme 
de  support  a  l'analyse  de  situation  qui  pourrait  supporter  le  travail  de  l'equipe  de  l'ORO: 
les  operateurs  et  leurs  superviseurs  et  l'ORO  lui-meme.  De  cette  presentation  des  divers 
niveaux  d'analyse  de  situation  qui  se  deroulent  dans  la  Salle  d'opdrations,  nous  concluons 
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que  la  modele  JDL  de  fusion  de  donnees  ne  peut  pas  etre  appliqud  directement  a  une 
equipe  organisde  hierarchiquement  et  devrait  etre  revisd  en  considdrant  la  notion 
d'analyse  de  situation. 

Finalement,  nous  presentons  nos  recommandations  pour  les  prochaines  etapes  k  rdaliser 
en  vue  de  la  creation  d'un  systeme  de  support  k  l'analyse  de  situation.  Plus 
specifiquement,  nous  reommandons  qu'une  approche  d'ingenierie  des  connaissances  soit 
realisee  en  conjonction  avec  une  analyse  cognitive  de  tSches  du  poste  d'ORO  et  des 
postes  des  membres-cle  de  son  equipe.  Cette  phase  d'ingenierie  des  connaissances  devrait 
permettre  aux  ingenieurs  de  la  connaissance  d'obtenir  des  modeles  formels  qui 
s'apparentent  aux  modeles  mentaux  et  des  images  mentales  qui  permettent  k  l'ORO  de 
former  sa  conscience  de  situation.  De  tels  modeles  formels  seront  utilises  pour  etablir  les 
principales  structures  de  donnees  et  de  connaissances  sur  lesquelles  un  systeme  de 
support  k  l'analyse  de  situation  pourra  etre  bati. 
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1 .  Introduction 

This  work  takes  place  in  the  context  of  a  long  range  project  that  aims  at  developing  a 
decision  support  system  (DSS)  for  Operation  Room  Officers  (OROs)  in  the  HALIFAX 
class  frigate.  Given  the  large  volume  of  incoming  data  from  various  sources,  the  time 
constraints  and  the  characteristics  of  the  activities  taking  place  in  the  Operation  Room, 
DREV  researchers  have  investigated  for  several  years  the  applicability  of  data  fusion 
techniques  for  the  development  of  a  DSS  for  OROs. 

The  JDL  data  fusion  model  (Waltz  and  Llinas  1990)  ordered  by  the  Joint  Directors  of 
Laboratories  Data  Fusion  Subpanel  is  a  framework  which  has  been  widely  used  to 
structure  the  main  activities  taking  place  in  the  data  fusion  domain.  The  revised  version 
of  the  JDL  model  (Steinberg  et  al.  1998)  aims  at  broadening  the  scope  of  the  model  so 
that  it  could  be  applied  to  both  civilian  and  military  application  domains.  Five  levels  of 
data  processing  are  distinguished:  Sub-object  assessment  (level  0),  object  assessment 
(level  1),  situation  assessment  (level  2),  impact  assessment  (level  3)  and  process 
refinement  (level  4).  Most  of  the  practical  systems  that  have  been  developed  in  the  past 
provide  functionalities  for  levels  0  and  1  of  the  JDL  model:  they  deal  with  the  difficult 
problems  of  multi-source  data  fusion.  Very  few  systems  provide  satisfactory  solutions  for 
levels  2,  3  and  4. 

For  several  years,  DREV  researchers  have  developed  systems  mainly  for  multisource 
data  fusion  (JDL  model's  levels  0  and  1).  When  working  on  situation  and  impact 
assessment  (levels  2  and  3),  they  faced  several  difficulties  and  discovered  that  one  main 
limitation  of  the  JDL  model  is  that  not  enough  attention  is  given  to  a  fundamental 
element  in  the  Command  and  Control  (C2)  process:  the  person  who  makes  decision.  The 
naval  application  domain  is  very  complex  and  developing  complete  automatic  solutions 
seems  out  of  reach  for  the  current  technology  and  may  be  undesirable.  In  several  military 
domains,  researchers  have  noticed  that  data  fusion  techniques  cannot  provide  on  their 
own  a  complete  and  satisfactory  solution  to  automate  the  complex  activities  taking  place 
in  C2  settings.  This  can  be  explained  by  the  fact  that  such  settings  involve  ill-structured 
problems,  uncertain  and  dynamic  environments,  conflicting,  shifting  or  ill-defined  goals, 
time  constraints,  high  stakes  and  pressure.  Given  the  current  technology  and  software 
development  know-how,  it  is  impossible  to  develop  a  data  fusion  system  that  would 
efficiently  integrate  all  the  functionalities  that  are  pictured  in  the  JDL  idealized  model 
without  considering  the  fundamental  contribution  of  human  operators.  Hence,  a  data 
fusion  system  should  be  thought  of  as  technological  means  to  support  human  decisions. 
How  can  such  a  system  be  useful  to  support  OROs'  decision  making  process?  During  the 
past  years  the  DREV  team  has  emphasized  the  importance  of  considering  the  cognitive 
aspects  of  C2  decision  making  processes,  and  more  specifically  of  providing  means  to 
develop  and  maintain  operators'  situation  awareness. 

Several  researchers  have  emphasized  the  key  role  played  by  situation  awareness  when 
making  decisions  in  complex  settings  (Endsley  1995).  From  the  operator's  point  of  view, 
situation  awareness  can  be  informally  defined  as  "knowing  what's  going  on  so  that  you 
can  figure  out  what  to  do".  In  the  context  of  the  project  for  developing  a  DSS  for  OROs 
in  the  HALIFAX  Class  Frigat,  a  Cognitive  Task  Analysis  of  the  ORO's  position 
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(Matthews  et  al.  1999)  has  been  recently  completed  and  provides  relevant  observations 
related  to  the  ORO's  cognitive  needs:  1)  Gaining  and  maintaining  situation  awareness  is  a 
key  issue  for  OROs;  2)  OROs  employ  a  variety  of  mental  pictures  and  mental  models  to 
achieve  their  cognitive  goals;  3)  Mission  preparation  has  great  significance  for 
establishing  mission  related  mental  models;  4)  Updating  situation  awareness  when 
coming  on  watch  is  critical  for  updating  ORO’s  mental  models. 

These  cognitive  aspects  of  the  ORO's  activities  cannot  be  accounted  for  in  the  JDL 
model.  However,  they  are  most  important  if  one  wants  to  develop  a  decision  support 
system  that  will  be  useful  to  OROs.  Hence,  it  seems  appropriate  to  adopt  a  wider 
perspective  by  examining  how  a  decision  support  system  can  take  advantage  of  data 
provided  by  a  data  fusion  system  and  present  them  in  a  way  that  fits  OROs'  mental 
models.  This  cognitive  fit  should  enable  OROs  to  gain  and  maintain  situation  awareness. 
In  this  context  the  Scientific  Authority  of  this  contract  proposed  us  to  investigate  the 
process  of  "situation  analysis".  Situation  analysis  can  be  defined  as  a  process  that  leads  a 
person  or  a  group  of  persons  to  a  state  of  situation  awareness. 

In  Section  2  we  mention  the  contract  objectives  and  give  an  overview  of  the 
accomplished  work. 

Section  3  presents  the  main  elements  of  the  widely-used  JDL  model  for  data  fusion  and 
discusses  its  limitations  when  considering  the  cognitive  aspects  of  the  ORO’s  activities. 
Researchers  have  shown  that  situation  awareness  plays  a  key  role  in  the  decision  making 
process  (Section  4).  A  recent  cognitive  task  analysis  for  the  ORO  position  has  been 
carried  out.  We  present  in  Section  5  its  main  conclusions  which  emphasize  the  key  role 
played  by  situation  awareness  and  mental  models  for  ORO’s  cognitive  activities. 

In  Section  6  we  introduce  the  notion  of  situation  analysis  as  it  was  defined  by  the 
Scientific  Authority  of  the  present  contract. 

In  Section  7  we  discuss  the  key  elements  that  influence  situation  analysis  in  the  context 
of  the  Operation  Room.  We  show  that  situation  analysis  is  a  broad  process  that  takes 
place  at  several  levels  at  once:  at  the  level  of  each  team  member  and  at  a  more  global 
level  which  corresponds  to  the  global  situation  understanding  of  the  team  taken  as  a 
whole. 

In  Section  8  we  propose  and  discuss  generic  architectures  for  a  situation  analysis  support 
system  that  could  be  used  to  support  the  work  of  the  ORO's  team:  operators  and  their 
supervisors  as  well  as  the  ORO  himself.  From  this  presentation  of  the  different  levels  of 
situation  analysis  taking  place  in  an  Operation  Room  we  conclude  that  the  JDL  Data 
Fusion  model  may  not  directly  apply  to  a  hierarchically  organized  team  and  should  be 
revised  considering  the  notion  of  situation  analysis. 

In  Section  9  we  present  our  recommendations  for  the  next  steps  to  be  performed  toward 
the  creation  of  a  situation  analysis  support  system.  More  specifically,  we  recommend 
that  a  knowledge  engineering  approach  be  coupled  with  the  cognitive  task  analysis  of  the 
ORO  and  the  key  members  of  his  team.  The  knowledge  engineering  phase  of  the  project 
should  enable  knowledge  engineers  to  obtain  formal  models  that  match  the  mental 
models  and  pictures  enabling  an  ORO  to  gain  and  maintain  situation  awareness.  Such 
formal  models  will  be  used  as  the  key  knowledge  and  data  structures  on  which  the 
situation  analysis  support  system  would  be  built. 
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2.  Contract  Objectives  and  Review  of  Accomplished  Work 

The  objectives  of  the  present  contract  are: 

-  To  survey  the  relevant  literature  in  relation  to  situation  analysis 

-  To  write  a  synthetic  report  on  the  subject  and  propose  a  definition  for  "situation 
analysis" 

-  To  propose  recommendations  for  an  object-oriented  analysis  of  a  situation  analysis 
support  system 

Here  are  the  main  activities  that  have  been  performed  in  order  to  reach  these  objectives: 

•  Literature  survey  using  documents  and  web  searches  on  the  following  areas: 

-  Data  fusion  process  (several  popular  DF  models  (JDL,  etc.);  an  example  in  the 

naval  domain;  advantages,  limitations;  use  of  artificial  intelligence  techniques) 

-  Situation  assessment  process 

-  Situation  awareness 

•  Study  of  the  cognitive  task  analysis  of  the  Halifax-Class  Operations  Room  Officer 

•  Study  of  several  papers  of  the  DREV  team  on  Data  Fusion  and  situation  awareness 

•  A  bibliography  base  has  been  built  using  Endnotes 

•  Preliminary  domain  synthesis  and  presentation  to  DREV  team 

•  Further  literature  study  and  refinement  of  the  domain  synthesis 

•  Report  writing  and  adjustments 
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3.  Data  Fusion 

Data  fusion  is  an  important  research  and  development  domain  for  both  military  and 
civilian  applications.  During  the  past  20  years  much  efforts  and  resources  have  been 
devoted  to  this  research  area  and  several  systems  have  been  built.  However,  most  of  these 
systems  do  not  fulfill  the  real  needs  of  their  users  which  are  often  flooded  by  data  coming 
from  various  sources  when  they  would  need  relevant  aggregated  information  that  could 
help  them  to  make  decisions.  The  JDL  data  fusion  model  ordered  by  the  Joint  Directors 
of  Laboratories  Data  Fusion  Subpanel  is  a  framework  which  has  been  widely  used  to 
structure  the  main  activities  taking  place  in  the  data  fusion  domain.  In  its  1992  version 
which  was  oriented  toward  military  applications,  the  JDL  model  distinguished  four  levels 
of  data  processing:  object  refinement,  situation  refinement,  threat  refinement  and  process 
refinement.  An  overview  of  this  model  is  presented  in  Appendix  1  of  the  present 
document. 

The  revised  version  of  the  JDL  model  (Steinberg  et  al.  1998)  aims  at  broadening  the 
scope  of  the  original  JDL  model  so  that  it  could  be  applied  to  both  civilian  and  military 
application  domains. 

A  general  concise  definition  of  the  data  fusion  process  is  given: 

Data  fusion  is  the  process  of  combining  data  to  refine  state  estimates  and  predictions. 

In  the  new  version  of  the  JDL  model,  five  levels  are  distinguished:  Sub-object  assessment 
(level  0),  object  assessment  (level  1),  situation  assessment  (level  2),  impact  assessment 
(level  3)  and  process  refinement  (level  4).  Let  us  comment  upon  those  levels  in  order  to 
emphasize  the  main  elements  that  are  of  importance  for  the  data  fusion  process 
(Steinberg  et  al.  1998). 

The  main  activities  of  level  0  aim  at  estimating  and  predicting  signal/object  observable 
states  on  the  basis  of  pixel/signal  level  data  association  and  characterization.  Such  a 
process  involves  hypothesizing  the  presence  of  a  signal  and  estimating  its  state. 

The  main  activities  of  level  1  aim  at  estimating  and  predicting  entity  states  on  the  basis  of 
observation-to-track  association,  continuous  state  estimation  (e.g.  kinematics)  and 
discrete  state  estimation  (e.g.  target  type  and  identification).  Such  a  process  involves 
associating  reports  (or  tracks  from  prioir  fusion  nodes  in  a  processing  sequence)  into 
association  hypotheses  (called  here  tracks).  Each  such  track  represents  the  hypothesis  that 
the  given  set  of  reports  is  the  total  set  of  reports  available  to  the  system  referencing  some 
individual  entity. 

The  main  activities  of  level  2  aim  at  estimating  and  predicting  relations  among  entities  to 
include  force  structure  and  cross  force  relations,  communications  and  perceptual 
influences,  physical  context,  etc.  Such  a  process  involves  associating  tracks  (i.e. 
hypothesized  entities)  into  aggregations.  The  state  of  the  aggregate  is  represented  by  a 
network  of  relations  among  its  elements.  As  the  class  of  estimated  relationships  and  the 
number  of  interrelated  entities  broaden,  people  use  the  term  situation  for  an  aggregate 
object  of  estimation. 


The  main  activities  of  level  3  aim  at  estimating  and  predicting  effects  on  situations  of 
planned  or  estimated/predicted  actions  by  the  participants;  to  include  interactions 
between  action  plans  of  multiple  players  (e.g.  assessing  susceptibilities  and 
vulnerabilities  to  estimated/predicted  threat  actions  given  one's  own  planned  actions). 
Such  a  process  involves  implementing  prediction  functions  drawing  particular  kinds  of 
inference  from  Level  2  associations.  Level  3  estimates  the  impact  of  an  assessed 
situation,  i.e.  the  outcome  of  various  plans  as  they  interact  with  one  another  and  with  the 
environment.  The  impact  estimate  can  include  likelihood  and  cost/utility  measures 
associated  with  potential  outcomes  of  a  player’s  planned  actions. 


Figure  1  Information  Workflow  among  JDL  model's  levels  (1998) 
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The  main  activities  of  level  4  aim  at  adaptatively  acquire  and  process  data  in  order  to 
support  mission  objectives.  It  is  part  of  the  Resource  Management  process.  This  involves 
assigning  tasks  to  resources. 

Figure  1  presents  an  information  workflow  across  the  levels  of  the  revised  version  of  the 
JDL  model. 

Another  important  remark  (Steinberg  et  al.  1998)  is  relative  to  the  importance  of 
estimating  the  value  of  entity  states  on  the  basis  of  the  context:  "A  system  that  integrates 
data  association  and  estimation  processes  of  all  levels  will  permit  entities  to  be 
understood  as  part  of  a  complex  situation". 

These  remarks  emphasize  situation  description  and  characterization  as  one  of  the  main 
goals  of  the  data  fusion  process.  However,  most  of  the  practical  systems  that  have  been 
developed  until  now  provide  functionalities  for  levels  0  and  1  of  the  JDL  model:  they 
deal  with  the  difficult  problems  of  multi-source  data  fusion.  Very  few  systems  provide 
satisfactory  solutions  for  levels  2,  3  and  4.  The  JDL  model  presents  an  ideal  view  of  the 
functionalities  of  a  data  fusion  system  that  should  be  able  to  support  every  activity  from 
data  acquisition  to  resource  management.  However,  building  such  a  system  is  beyond  the 
capability  of  current  technology  and  software  development  methods.  Consequently,  we 
can  only  view  a  data  fusion  system  as  a  system  supporting  an  operator  who  makes 
decisions  and  often  needs  to  orient  several  activities  of  the  data  fusion  system.  Hence, 
one  of  the  main  limitations  of  the  JDL  model  is  that  not  enough  attention  is  given  to  the 
person  who  makes  decision. 

4.  Situation  Awareness 

Given  the  complexity  of  situations  encountered  in  C2  activities,  it  has  been  observed  by 
numerous  researchers  that  decision  makers  can  operate  efficiently  if  they  can  get  a  global 
understanding  of  the  attended  situations.  Discovery  of  relationships  between  objects,  as 
well  as  patterns  or  configurations  of  objects  may  lead  the  decision  maker  to  infer  the 
intents  of  involved  player's  and  act  accordingly.  Situations  being  highly  dynamic  in  most 
cases,  decision  makers  must  also  be  able  to  follow  the  evolution  of  attended  situations. 
All  these  observations  lead  to  the  notion  of  situation  awareness.  Here  is  a  general 
definition  of  SA  (Endsley  1987;  Endsley  1988): 

Situation  awareness  is  the  perception  of  the  elements  in  the  environment  within  a 
volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection 
of  their  status  in  the  near  future. 

It  is  necessary  to  distinguish  the  term  situation  awareness  (SA)  as  a  state  of  knowledge, 
from  the  processes  used  to  achieve  that  state.  These  processes,  which  may  vary  widely 
among  individuals  and  contexts,  will  be  referred  to  as  situation  assessment  or  as  the 
process  of  achieving,  acquiring,  or  maintaining  SA. 

SA  as  defined  here  does  not  encompass  all  of  a  person’s  knowledge.  It  refers  to  only  that 
portion  pertaining  to  the  state  of  a  dynamic  environment.  Established  doctrine,  rales, 
procedures,  checklists,  and  the  like  -  though  important  and  relevant  to  the  decision¬ 
making  process  -  are  fairly  static  knowledge  sources  that  fall  outside  the  boundaries  of 
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the  term.  In  addition,  SA  is  explicitly  recognized  as  a  construct  separate  from  decision 
making  and  performance.  SA,  decision  making,  and  performance  are  different  stages  with 
different  factors  influencing  them  and  with  wholly  different  approaches  for  dealing  with 
each  of  them;  thus  it  is  important  to  treat  these  constructs  separately.  Attention,  working 
memory,  workload,  and  stress  are  all  related  constructs  that  can  affect  SA  but  that  can 
also  be  seen  as  separate  from  it. 

Figure  2  provides  a  basis  for  discussing  SA  in  terms  of  its  role  in  the  overall  decision¬ 
making  process.  According  to  this  model,  a  person’s  perception  of  the  relevant  elements 
in  the  environment  from  system  displays  or  directly  by  the  senses,  forms  the  basis  for  his 
or  her  SA.  Details  of  Endsley's  model  are  given  in  Appendix  2. 


Figure  2:  Endsley's  model  of  situation  awareness 

SA  can  be  shown  to  be  important  in  a  variety  of  contexts:  aircraft,  air  traffic  control, 
large-systems  operations,  tactical  and  strategic  systems  and  many  other  everyday 
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activities  necessitate  a  dynamic  update  of  the  situation  to  function  (Zsambok  1997; 
Zsambok  and  Klein  1997). 

The  need  for  SA  applies  in  a  wide  variety  of  environments.  Acquiring  and  maintaining 
SA  becomes  increasingly  difficult.  Because  the  state  of  the  environment  is  constantly 
changing,  often  in  complex  ways,  a  major  portion  of  the  operator’s  job  becomes  that  of 
obtaining  and  maintaining  good  SA.  Operators  must  do  more  than  simply  perceive  the 
state  of  their  environment.  They  must  understand  the  integrated  meaning  of  what  they  are 
perceiving  in  light  of  their  goals.  Researchers  in  many  areas  have  found  that  expert 
decision  makers  will  act  first  to  classify  and  understand  a  situation,  immediately 
proceeding  to  action  selection.  There  is  evidence  that  an  integrated  picture  of  the  current 
situation  may  be  matched  to  prototypical  situations  in  memory,  each  prototypical 
situation  corresponding  to  a  “correct”  action  or  decision.  Experts  use  pattern- matching 
mechanisms  to  draw  on  long-term  memory  structures  (such  as  schemas  and  scripts)  that 
allow  them  to  quickly  understand  a  given  situation. 

A  formal  definition  of  situation  awareness  is  included  in  several  US  Navy  specifications. 
It  says:  "Operator  SA  is  comprised  of  detecting  information  in  the  environment, 
processing  the  information  with  relevant  knowledge  to  create  a  mental  picture  of  the 
current  situation,  and  acting  on  this  picture  to  make  a  decision  or  explore  further". 

Showing  the  importance  of  such  issues,  the  Situational  Awareness  Integrated  Product 
Team  (SA  IPT)  has  been  formed  in  the  US  in  response  to  fleet  tactical  aircraft  aviators 
ranking  situation  awareness  as  a  critical  mission  concern  (Garner  and  Assenmacher 
1997).  SA  IPT  held  several  symposia  in  order  to  provide  a  forum  for  information 
exchanges  about  situation  awareness  between  academia,  military  researchers  and  industry 
(references).  We  can  conjecture  that  work  on  situation  awareness  will  have  an  important 
influence  on  the  data  fusion  community  in  the  coming  years.  This  could  lead  researchers 
to  reconsider  the  role  of  data  fusion  in  the  context  of  a  broader  view  of  the  decision 
making  process  based  on  the  ability  of  the  operator  to  gain  and  maintain  situation 
awareness. 

5.  A  Cognitive  Task  Analysis  of  the  ORO's  Position  (1999) 

The  naval  application  domain  is  no  exception.  A  Cognitive  Task  Analysis  (CTA)  of  the 
ORO's  position  has  been  recently  completed  (Matthews  et  al.  1999)  and  shows  the 
importance  of  situation  awareness  for  these  decision  makers.  We  sum  up  in  this  section 
the  main  observations  of  the  CTA  that  are  relevant  for  our  current  discussion. 

This  CTA  provides  relevant  observations  related  to  the  ORO's  cognitive  needs.  We 
mention  here  some  of  the  main  conclusions  of  this  study.  Gaining  and  maintaining 
situation  awareness1  is  a  key  issue  for  OROs.  OROs  employ  a  variety  of  mental  pictures 
and  mental  models  to  achieve  their  cognitive  goals.  Mission  preparation  has  great 


1  A  model  of  situation  awareness  has  been  proposed  by  Endsley  (Endsley  1995)  who  distinguishes  three 
main  activities  for  gaining  awareness:  perception  of  current  elements  of  a  situation,  comprehension  of 
current  situation  and  projection  of  future  status. 
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significance  for  establishing  mission  related  mental  models.  Updating  situation  awareness 
when  coming  on  watch  is  critical  for  updating  ORO's  mental  models. 

For  situation  awareness  several  major  functional  requirements  were  identified. 
Information  systems  should  give  less  data  and  more  "information".  There  is  a  need  for 
information  acquisition  and  integration.  It  is  important  to  have  a  cognitive  match  between 
the  available  data  and  the  user's  mental  model(s).  A  system  should  help  OROs  to  regain 
awareness  after  switches  in  attention  between  different  areas  of  focus,  and  provide  alerts 
to  significant  changes  of  unattended  areas  of  the  situation. 

Background  tasks  (such  as  maintaining  situation  awareness  of  the  evolving  operation  or 
dealing  with  incoming  text  messages)  can  be  differentiated  from  foreground  or  threat- 
related  tasks.  OROs  need  to  be  able  to  switch  between  foreground  and  background  tasks 
with  seamless  integration  of  data.  A  major  ORO  function  is  to  manage  the  overall 
Operation  Room  team's  threat  response  rather  than  to  be  directly  involved  in  details  of 
responding  to  particular  threats.  Common  implicit  intent  and  understanding  among 
Operation  Room  team  members  has  particular  significance  for  communication 
effectiveness. 

Hence,  this  cognitive  task  analysis  not  only  shows  the  importance  of  situation  awareness 
in  the  ORO's  decision  making  process,  but  it  also  emphasizes  the  fact  that  OROs  use 
mental  models  and  pictures  in  order  to  get  an  understanding  of  the  attended  situation. 


Figure  3:  An  initial  presentation  of  Situation  Analysis 

6.  Situation  Analysis:  Initial  Definition 

During  the  past  years  the  DREV  team  has  investigated  several  aspects  of  the  notion  of 
situation  awareness  and  its  impact  on  the  development  of  data  fusion  systems  (Paradis  et 
al.  1997a;  Paradis  et  al.  1997b;  Roy  and  Bosse  1998).  They  found  out  that  the  cognitive 
aspects  of  the  ORO's  activities  cannot  be  accounted  for  in  the  JDL  model  and  that 
mechanisms  should  be  integrated  in  data  fusion  systems  in  order  to  help  users -to  get  and 
maintain  situation  awareness.  This  led  the  Scientific  Authority  of  this  contract  to  propose 
the  study  of  the  process  of  Situation  Analysis. 
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Situation  analysis  can  be  defined  as  the  process  that  allows  a  person  to  gather  and 
structure  all  the  elements  and  their  relations  that  allow  him  or  her  to  mentally 
understand  a  specific  situation  and  to  be  aware  of  its  potential  consequences. 

Figure  3  presents  a  global  view  of  such  a  process.  This  view  emphasizes  the  fact  that 
situation  analysis  should  help  the  decision  maker  to  increase  his  or  her  degree  of  situation 
awareness  during  the  whole  process  of  situation  analysis.  Situation  analysis  starts  from 
the  information  obtained  through  perception  and  gathered  into  a  preliminary  object 
model.  Then,  a  sub-process  called  situation  modeling  transforms  the  preliminary  object 
model  into  a  generic  situation  model.  Finally  a  sub-process  of  situation  and  threat 
assessment  derives  a  threat  model  from  the  generic  situation  model. 

This  view  also  emphasizes  the  primary  role  played  by  models:  the  preliminary  object 
model  and  the  generic  situation  model.  It  can  be  expected  that  a  system  will  be  an 
efficient  aid  to  gain  and  sustain  situation  awareness  thanks  to  the  composition  of  these 
models  and  the  way  the  associated  information  will  be  displayed  to  the  user.  Hence,  the 
importance  of  the  sub-process  of  situation  modeling  which  manipulates  these  models.  In 
the  rest  of  this  report  we  will  focus  on  the  main  elements  that  can  help  us  further 
understand  the  process  of  situation  analysis  and  its  influence  on  the  development  of  a 
decision  support  system  for  OROs. 


Mainlv  JDL  levels  0  &  1 


Figure  4:  An  overview  of  the  main  interactions  taking  place  in  the  Operation  Room 
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7.  The  Context  of  Situation  Analysis 

As  a  starting  point  let  us  present  an  overview  of  the  interactions  that  take  place  in  the 
Operation  Room:  interactions  between  operators,  existing  systems  and  the  ORO.  Figure  4 
displays  a  diagram  which  emphasizes  those  interactions. 

The  Operation  Room  already  contains  several  systems  that  gather  various  data  about  the 
environment.  These  systems  correspond  to  JDL  Model's  levels  0  and  1.  They  display 
these  data  to  operators  that  can  perform  actions  in  order  to  monitor  the  activities  of  the 
systems  those  systems.  In  Figure  4  the  dashed  arrows  symbolize  the  operators'  perceptual 
activities.  Operators  interact  together  as  well  as  with  the  ORO.  As  a  simplification,  we 
did  not  draw  on  this  diagram  the  roles  of  supervisors  who  supervise  a  group  of  operators. 
However,  supervisors  would  have  with  operators  similar  interactions  as  those  displayed 
between  ORO  and  operators  in  Figure  4.  Whenever  necessary,  the  ORO  (or  a  group 
supervisor)  can  go  and  stand  besides  an  operator  and  look  at  specific  data  on  his  or  her 
screen.  The  ORO  mentally  elaborates  a  mental  picture  of  the  situation  by  directly 
observing  data,  memorizing  information  obtained  from  various  reports,  gathering 
information  from  operators  and  their  supervisors  and  using  his  experience  and  intuition. 
This  is  the  way  the  ORO  currently  creates  and  maintain  situation  awareness. 


Figure  5:  Main  elements  involved  in  an  operator's  analysis  of  the  situation 
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We  should  emphasize  that  every  member  of  the  ORO's  team  develops  and  maintains  an 
awareness  of  the  situation.  However,  an  individual’s  situation  awareness  usually  differs 
from  that  of  another  individual  for  several  obvious  reasons:  differences  of  the  data 
monitored  by  each  individual  given  his  or  her  role  and  specialty;  difference  of  their 
mental  models,  difference  of  their  responsibilities  in  the  team  which  results  in  the 
attendance  on  different  aspects  of  the  situation. 

Figure  5  emphasizes  the  main  elements  that  characterize  an  operator's  activities  in  the 
Operation  Room.  We  distinguish  3  categories  of  activities:  system  monitoring  activities, 
operator's  cognitive  tasks  (reasoning,  decision  making,  mental  simulations,  etc.)  and 
interactions  with  other  team  members.  It  is  clear  that  all  these  activities  may  be  carried 
out  concurrently.  The  main  point  here  is  that  these  activities  enable  the  operator  to 
analyze  the  attended  situation  and  to  gain  situation  awareness,  hence  creating  and 
maintaining  mental  models  and  mental  pictures.  Obviously,  the  operator’s  knowledge, 
skills  and  training  influence  the  way  he  or  she  acts  and  consequently  his  or  her 
understanding  of  the  situation. 


Figure  6:  Main  elements  involved  in  an  ORO's  analysis  of  the  situation 

Figure  6  presents  the  main  elements  that  characterize  an  ORO's  activities  in  the  Operation 
Room.  We  distinguish  3  categories  of  activities;  system  monitoring  activities,  ORO's 
cognitive  tasks  (reasoning,  decision  making,  mental  simulations,  etc.)  and  interactions 
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with  other  team  members.  The  ORO's  system  monitoring  activities  refer  to  his  use  of 
phone,  reading  reports,  attending  data  on  screens,  etc.  The  ORO's  cognitive  activities 
correspond  to  creating  mental  models  and  mental  pictures  of  the  situation,  reasoning, 
decision  making,  performance  of  mental  simulations,  etc.  The  ORO's  interaction 
activities  refer  to  the  various  interactions  that  take  place  with  team  members  in  order  to 
get  information,  to  give  orders  and  ask  for  advice. 

It  should  be  clear  now  that  situation  analysis  is  a  broad  process  that  takes  place  at  several 
levels  at  once:  at  the  level  of  each  team  member  and  at  a  more  global  level  which 
corresponds  to  the  global  situation  understanding  of  the  team  taken  as  a  whole.  Ideally, 
the  ORO  is  the  person  who  would  embody  this  global  understanding  of  the  situation. 
Hence,  when  studying  the  process  of  situation  analysis,  researchers  and  developers 
should  be  aware  of  these  individual  differences  of  situation  understanding  and  awareness. 

8.  Toward  a  System  for  Supporting  Situation  Analysis 

8.1  Introduction 

The  preceding  discussion  sheds  new  light  on  the  current  limitations  of  data  fusion 
systems.  We  already  mentioned  that  the  JDL  model  presents  an  ideal  view  of  the 
functionalities  of  a  data  fusion  system  that  should  be  able  to  support  every  activity  from 
data  acquisition  to  resource  management.  In  practice,  researchers  and  developers  have 
difficulties  to  deal  with  the  complexity  of  the  activities  involved  in  such  a  process. 

This  is  not  a  surprise  because  there  are  numerous  activities  addressing  different  levels  of 
decision:  operational,  tactical  and  strategic.  To  use  an  analogy,  developing  such  a  data 
fusion  system  would  be  equivalent  to  designing  all  the  systems  of  a  company  at  once: 
operational  systems  that  deal  with  billing,  stock  management,  sales  management,  etc.,  as 
well  as  decision  support  systems  for  managers  and  a  decision  support  system  for 
executives.  It  is  impossible  to  develop  all  these  systems  at  once  because  operational 
systems  provide  aggregated  data  that  are  used  by  tactical  decision  support  systems  that  in 
turn  provide  information  to  executive  decision  support  systems. 

In  an  Operation  Room,  there  are  systems  that  are  used  for  different  kinds  of  warfare 
areas:  anti-air  warfare,  anti-surface  warfare,  anti-submarine  warfare,  mine  warfare  and 
command  and  control  warfare.  Each  of  these  areas  is  attended  by  one  or  several  persons 
of  the  ORO's  team.  Each  person  gains  situation  awareness  in  his  or  her  own  domain  of 
competence  and  the  ORO  gains  awareness  of  the  global  situation.  Hence,  the  current 
systems  (mainly  JDL  model’s  levels  0  and  1)  that  provide  data  to  the  various  specialists 
of  the  ORO's  team  are  operational  systems. 

In  order  to  develop  a  useful  decision  support  system  for  sub-team  supervisors  in  charge 
of  specific  areas  (tactical  systems),  we  need  to  identify  which  are  the  relevant  elements 
that  compose  their  understanding  of  the  situation.  Hence,  these  decision  support  systems 
will  provide  a  partial  view  of  the  situation  that  fits  the  cognitive  needs  of  these 
supervisors. 
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Considering  the  perspective  of  the  ORO  who  must  make  strategic  decisions  in  the 
Operation  Room,  we  can  say  that  he  has  the  same  position  as  a  company's  executive. 
Hence,  the  decision  support  system  that  will  provide  an  ORO  with  useful  information 
will  need  to  aggregate  the  models  of  the  specific  situations  provided  by  the  decision 
support  systems  used  by  the  sub-team  supervisors  in  charge  of  specific  areas.  This 
integrated  view  of  the  situation  should  fit  the  ORO's  cognitive  needs  (mental  models  and 
pictures)  in  order  to  support  his  decision  making  process. 


Consequently,  the  process  of  situation  analysis  must  take  into  account  the  level  of 
decision  that  characterizes  the  users  of  the  decision  support  system  to  be  built.  We 
distinguish  the  operational  level  (operators'  activities),  the  tactical  level  (activities  of 
operators'  supervisors)  and  the  strategic  level  (ORO's  activities).  Situation  analysis  is 
different  at  each  of  these  levels.  Tactical  situation  models  will  be  built  on  the  basis  of 
operational  situation  models  and  the  strategic  situation  model  will  be  built  on  the  basis  of 
the  tactical  situation  models. 


Mainly  JDL  levels  2  &  3 


Figure  7:  A  generic  architecture  of  an  operator’s  situation  analysis  support  system 


8.2  Situation  Analysis  at  the  Operational  Level 


At  the  operators’  operational  level,  a  situation  analysis  support  system  (JDL  model's 
levels  2,  3  and  4)  would  have  to  aggregate  data  obtained  from  multi-sensor  data  fusion 
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systems  (JDL  model's  levels  0  and  1)  into  a  model  of  the  situation  which  emphasizes  the 
elements  needed  by  the  operator  to  conduct  the  operations  of  the  system  in  its  specific 
area. 

Figure  7  presents  a  generic  architecture  of  a  situation  analysis  support  system  that  could 
be  used  by  an  operator.  The  embedding  rectangle  symbolizes  the  situation  analysis 
support  system  (SASS).  The  objective  of  this  figure  is  to  illustrate  some  principles  that 
may  be  useful  to  the  designers  of  future  situation  analysis  systems. 

•  The  operator  is  not  in  direct  contact  with  the  control  room  systems  any  more  as  he 
was  in  the  Operation  Room's  current  configuration  (Figure  4).  In  Figure  7  we  see 
that  the  SAS's  modules  monitor  the  control  room  systems  (corresponding  to  JDL 
levels  0  and  1 ).  As  displayed  in  Figure  7,  a  single  SASS  can  monitor  several  control 
room  systems.  We  suggest  that  these  control  room  systems  record  their  data  in  data 
bases  that  can  be  accessed  by  the  SAS's  modules. 

•  The  objective  of  the  SASS  is  to  build  an  image  of  the  situation  (called  situation 
display  in  Figure  7)  which  fits  with  the  operator's  cognitive  needs.  This  situation 
display  should  correspond  to  the  operator's  mental  models  or  at  least  provide 
relevant  information  so  that  the  operator  will  be  able  to  build  his  or  her  mental 
models  of  the  situation. 

•  We  distinguish  three  kinds  of  modules  in  the  SASS's  architecture  in  order  to 
emphasize  the  fact  that  only  certain  activities  can  be  automated.  We  distinguish 
three  categories  of  tasks  according  to  the  operator's  familiarity  with  them:  routine 
tasks  that  can  be  automated,  familiar  tasks  that  cannot  be  completely  automated  but 
are  known  by  the  operator  and  unfamiliar  tasks  that  the  operator  will  discover  during 
a  course  of  action.  It  should  be  clear  that  we  consider  here  all  the  tasks  that  the 
operator  might  have  to  perform  when  being  on  watch  in  the  Operation  Room. 
Although  only  some  of  these  tasks  can  be  automated,  the  SASS  modules  should  be 
able  to  provide  the  operator  with  relevant  information  so  that  he  or  she  can  build  or 
update  his  or  her  mental  model  of  the  situation,  assess  the  situation  and  choose  an 
appropriate  course  of  actions.  Hence,  each  module  of  the  SASS  contributes  to  the 
creation  and  update  of  the  situation  display. 

•  Let  us  remark  that  the  three  kinds  of  modules  can  be  distributed  in  different  ways  in 
a  real  system  architecture.  This  distribution  depends  on  the  category  of  architecture 
that  is  chosen  (expert  system,  blackboard  system,  multi-agent  system,  etc.)  by  the 
designers. 

•  Automatic  tasks  correspond  to  routine  activities  described  by  procedures  that  are 
well  known  and  can  be  automated.  A  module  implementing  automatic  tasks  is  able 
to  automatically  monitor  certain  functionalities  of  the  control  room  systems.  Usually 
it  will  send  reports  to  the  controller  and  require  approval  for  certain  actions,  when 
deemed  necessary  by  the  system  designers.  The  information  gathered  by  modules 
implementing  automatic  tasks  contributes  to  the  creation  and  update  of  the  situation 
display. 

•  Modules  supporting  familiar  tasks  partly  automate  tasks  that  are  known  by  the 
operator;  those  tasks  requiring  the  operator’s  involvement  for  actions  such  as  choice 
among  several  possible  alternatives,  decisions  with  incomplete  information, 
orientation  of  data  acquisition  systems,  etc.  One  important  role  of  these  modules  is 
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to  contribute  to  the  creation  and  update  of  the  situation  display.  The  situation  display 
must  present  information  in  a  way  that  will  facilitate  the  operator's  decision  making 
process.  Hence,  the  importance  of  structuring  and  displaying  information  about  the 
situation  in  a  way  that  fits  the  operator's  mental  models.  Since  these  tasks  are 
familiar,  system  designers  may  implement  in  the  module  certain  functionalities 
(rules  for  example)  that  will  be  able  to  provide  advice  to  the  operator  with  resepct  to 
the  attended  situation. 

•  Modules  supporting  unfamiliar  tasks  can  only  gather  information  and  display  it  in  a 
way  that  will  enable  the  operator  to  understand  the  situation  and  help  him  or  her  to 
make  decisions.  It  is  difficult  to  design  such  modules  because  of  the  unfamiliarity  of 
the  tasks.  One  way  to  deal  with  this  difficulty  is  to  try  to  implement  generic 
knowledge  structures  that  will  be  able  to  record  any  kind  of  fact  characterizing 
events  occurring  in  the  environment.  Again,  one  important  role  of  these  modules  is 
to  contribute  to  the  creation  and  update  of  the  situation  display.  The  situation  display 
must  present  information  in  a  way  that  will  facilitate  the  operator's  decision  making 
process  when  dealing  with  unfamiliar  tasks. 

•  The  proposed  generic  architecture  also  provides  ways  to  make  the  system  evolve.  If 
system  designers  implement  functionalities  that  will  record  operator's  actions  along 
with  the  situation  states,  they  will  be  able  to  analyze  real  courses  of  actions.  Such 
analyses  may  enable  designers  to  better  understand  familiar  tasks  and  automate 
certain  of  them.  In  the  same  way  they  will  be  able  to  analyze  the  operators  actions 
when  dealing  with  unfamiliar  tasks  and  enhance  modules  supporting  familiar  tasks 
accordingly. 

8.3  Situation  Analysis  at  the  Tactical  and  Strategic  Levels 

Now,  let  us  look  at  the  tactical  level  (activities  of  operators'  supervisors)  and  the  strategic 
level  (ORO's  activities).  At  these  levels,  operators'  supervisors  and  the  ORO  do  not  deal 
with  operational  data,  but  with  aggregated  information.  As  we  previously  mentioned, 
they  are  in  the  same  position  as  managers  and  executives  in  a  company.  Hence,  a 
situation  analysis  support  system  (SASS)  for  operators'  supervisors  mainly  takes 
information  about  the  attended  situation  in  the  knowledge  bases  and  data  bases  of  the 
operators'  SASSs.  The  supervisor's  SASS  modules  will  aggregate  and  manipulate 
information  and  knowledge  provided  by  the  operators'  SASSs  in  order  to  build  an  image 
of  the  situation  display)  that  fits  the  supervisor's  mental  models  and  pictures. 

Figure  8  presents  a  generic  architecture  which  illustrates  how  a  SASS  for  an  officer  (an 
operators'  supervisor  in  our  case)  can  be  developed  on  top  of  operators'  SASSs.  We  have 
not  detailed  the  modules  composing  the  officer's  SASS,  because  they  are  domain 
dependent.  We  do  not  know  if  the  distinction  that  we  made  for  the  operator's  SASS 
between  routine,  familiar  and  unfamiliar  tasks  still  applies  to  the  officer's  S  AS. 

In  Figure  8  we  also  distinguish  the  officer's  SASS  from  the  decision  support  system. 
Obviously,  they  are  part  of  the  same  system  that  helps  the  officer  to  understand  the 
situation  and  support  decision  making.  We  made  this  distinction  in  order  to  make  clear 
the  fact  that  a  situation  analysis  support  system  can  stand  alone  without  providing  advice 
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to  the  user  with  respect  to  the  actions  that  can  be  performed  in  order  to  deal  with  the 
attended  situation. 


The  same  kind  of  architecture  applies  to  the  strategic  level.  Hence,  an  ORO's  SASS 
would  take  its  basic  information  about  the  situation  into  the  knowledge  bases  and  data 
bases  of  the  officers'  SASSs.  The  ORO's  SASS  modules  would  aggregate  and  manipulate 
information  and  knowledge  provided  by  the  officers'  SASs  in  order  to  build  an  image  of 
the  situation  (situation  display)  that  fits  the  ORO's  mental  models  and  pictures. 


In  this  section  we  presented  generic  system  architectures  that  characterize  a  situation 
analysis  support  system.  We  showed  that  a  SASS  architecture  will  be  different  depending 
on  the  level  of  situation  analysis  that  we  want  to  automate.  We  distinguished  the 
operational  level  (operators'  activities),  the  tactical  level  (activities  of  operators' 
supervisors)  and  the  strategic  level  (ORO's  activities). 
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Situation  analysis  is  different  at  each  of  these  levels.  Operational  SASSs  obtain  their  data 
from  the  control  room  systems  (JDL  model's  levels  0  and  1)  and  build  a  situation  model 
that  fits  the  operator's  cognitive  needs.  Tactical  situation  models  are  built  on  the  basis  of 
operators’  situation  models  and  the  strategic  situation  model  are  built  on  the  basis  of  the 
tactical  situation  models. 

An  interesting  conclusion  of  this  presentation  of  the  different  levels  of  situation  analysis 
taking  place  in  an  Operation  Room  is  that  the  traditional  Data  Fusion  model  (the  five 
levels  of  the  revised  version  of  the  JDL  model)  may  not  directly  apply  to  a  hierarchically 
organized  team.  It  appears  clearly  in  Figure  7  that  the  operator's  situation  analysis  support 
system  monitors  the  control  room  systems  (JDL  model's  levels  0  and  1).  The  operator's 
situation  analysis  support  system  mainly  covers  levels  2  and  3  of  the  JDL  model 
corresponding  to  situation  assessment  and  impact  assessment.  Resource  management 
(JDL  level  4)  takes  place  at  another  level  (tactical  or  strategic  level). 

In  Figure  8  it  appears  clearly  that  the  Officer's  SASS  is  not  in  contact  with  the  control 
room  systems  (JDL  model's  levels  0  and  1).  The  officer's  situation  analysis  support 
system  covers  levels  2,  3  and  4  of  the  JDL  model  corresponding  to  situation  assessment 
and  impact  assessment  and  resource  management.  It  is  clear  that  only  a  part  of  resource 
management  is  dealt  with  at  the  officer's  tactical  level.  It  is  also  clear  that  the  same 
considerations  apply  to  the  strategic  level  where  the  ORO's  SASS  covers  levels  2,  3  and  4 
of  the  JDL  model. 

Another  important  conclusion  of  this  discussion  is  that  the  various  team  members  do  not 
have  the  same  understanding  of  the  situation,  given  the  level  of  their  interventions  and 
responsibilities.  Operators  have  a  specific  view  of  the  situation  whereas  the  ORO  should 
have  a  global  understanding  of  the  situation. 

We  think  that  the  initial  definition  of  Situation  Analysis  proposed  Scientific  Authority  of 
this  contract  needs  no  change  in  order  to  take  into  account  the  observations  made  during 
the  present  study. 

Situation  analysis  is  the  process  that  allows  a  person  to  gather  and  structure  all  the 
elements  and  their  relations  that  allow  him  or  her  to  mentally  understand  a  specific 
situation  and  to  be  aware  of  its  potential  consequences. 

We  may  suggest  some  complements  to  that  definition. 

Situation  analysis  is  a  cognitive  process  that  should  not  be  confused  with  the 
computer-based  process  of  data  fusion. 

A  situation  analysis  support  system  is  a  software  that  supports  a  person's  situation 
analysis  and  should  provide  a  situation  display  that  fits  this  person's  mental  models 
and  pictures  which  are  relevant  to  the  attended  situation. 

If  the  person  works  at  an  operational  level,  the  data  that  will  be  fed  into  his  or  her 
situation  analysis  support  system  will  come  from  a  data  fusion  system.  If  the  person 
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works  at  a  tactical  or  a  strategic  level,  the  data  that  will  be  fed  into  his  or  her 
situation  analysis  support  system  will  not  come  from  a  data  fusion  system  but  from 
another  situation  analysis  support  system  of  a  lower  level  ( operational  or  tactical). 


9.  Recommendations  for  an  Object-Oriented  Analysis  of  a  Situa¬ 
tion  Analysis  Support  System 

9.1  Preliminary  Discussion 

Given  the  preceding  discussion,  it  seems  too  early  to  give  precise  recommendations  for 
an  object  oriented  analysis  of  a  SASS.  We  do  not  have  enough  information  about  the 
tasks  carried  out  by  the  different  members  of  the  ORO's  team,  including  the  ORO 
himself.  We  know  that  other  researchers  are  currently  performing  work  and  task  analyses 
of  the  ORO’s  position.  We  recommend  that  the  DREV  team  wait  the  results  of  these 
analyses  in  order  to  better  plan  and  coordinate  further  activities.  However,  we  can 
identify  several  activities  that  could  be  carried  out  in  the  future  in  order  to  prepare  the 
analysis  and  design  of  a  SASS. 

To  begin  with,  we  recommend  that  the  DREV  team  examine  the  results  of  the  work  and 
task  analyses  of  the  ORO's  position  in  order  to  determine  which  data  sources  are  used  by 
an  ORO  in  order  to  build  his  mental  models  of  the  attended  situation.  We  conjecture  that 
an  important  part  of  these  data  are  provided  by  operators  and/or  their  supervisors.  If  this 
conjecture  is  verified  by  the  work  and  task  analyses  currently  performed,  then  the  generic 
architectures  that  we  presented  in  Figures  7  and  8  will  apply  to  the  ORO's  team.  Hence, 
the  DREV  team  will  have  to  decide  which  level  of  situation  analysis  will  be  examined  in 
the  next  phases  of  the  project. 

We  showed  in  Section  8  that  it  would  be  quite  difficult  to  try  to  develop  a  SASS  for  the 
ORO  right  on  top  of  a  data  fusion  system  of  level  0  and  1  according  to  the  JDL  model.  It 
seems  more  reasonable  to  develop  a  SASS  for  operators  on  top  of  these  Data  fusion 
systems.  Figure  9  presents  a  general  architecture  of  a  system  that  the  DREV  team 
proposed  to  support  the  levels  2  and  3  of  the  JDL  model  on  top  of  the  multisource  data 
fusion  systems  RM  and  MSDF  (levels  0  and  1  of  the  JDL  model).  We  think  that  a  SASS 
for  the  Operation  Room's  operators  would  have  similar  functionalities  and  would  also  be 
able  to  display  the  attended  situation  in  a  way  that  fits  the  operator’s  mental  model.  The 
architecture  proposed  in  Figure  9  should  be  modified  in  order  to  show  the  data  structures 
which  contain  the  description  of  the  various  elements  of  the  situation.  This  is  important 
because  these  structures  will  provide  the  basic  data  that  will  be  used  by  the  SASS 
designed  for  the  operator's  supervisors  (and  for  the  ORO). 

9.2  Coupling  Cognitive  Task  Analysis  and  Knowledge  Engineering 

Situation  Analysis  aims  at  allowing  a  person  to  mentally  understand  a  specific  situation 
and  to  be  aware  of  its  potential  consequences.  Research  works  on  situation  awareness 
have  shown  that  people  rely  on  the  use  of  mental  models  when  gaining  and  maintaining 
situation  awareness.  Endsley  (Endsley  1995)  and  various  other  authors  (Zsambok  and 
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Klein  1997)  underline  the  importance  of  schemas  and  scripts  that  structure  a  person's 
long  term  memory  and  contribute  to  the  creation  of  his  or  her  mental  models  of  the 
situation.  In  the  application  domain  of  our  study,  (Matthews  et  al.  1999)  recently 
completed  a  cognitive  task  analysis  of  the  ORO's  position  in  the  Operation  Room.  They 
observed  that  OROs  employ  a  variety  of  mental  pictures  and  mental  models  in  order  to 
achieve  their  cognitive  goals  and  to  gain  and  maintain  situation  awareness.  From  all  these 
observations  we  can  conclude  that  identifying  the  content  of  a  person's  mental  models 
relative  to  situations  of  interest  is  a  crucial  issue  when  developing  a  SASS. 


Figure  9:  A  generic  Situation  and  Threat  Assessment  System  (DREV  proposal) 
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The  data  structures  that  represent  the  elements  of  a  situation  and  their  relationships  are  at 
the  heart  of  a  SASS.  These  data  structures  should  allow  the  SASS  to  propose  to  a  user  a 
situation  display  that  fits  his  or  her  mental  models  and  pictures  relative  to  the  attended 
situation.  In  order  to  be  able  to  create  those  data  structures,  knowledge  engineers  should 
be  able  to  identify  which  elements  make  up  the  mental  models  of  the  users  of  the  SASS 
to  be  built. 

In  the  context  of  the  creation  of  a  SASS  for  OROs,  it  will  be  necessary  to  obtain  formal 
models  that  match  the  mental  models  and  pictures  enabling  an  officer  to  gain  and 
maintain  situation  awareness.  Such  formal  models  will  be  used  as  the  key  knowledge  and 
data  structures  on  which  the  SASS  will  be  built.  Hence,  it  is  recommended  to  carry  out  a 
cognitive  task  analysis  coupled  with  a  knowledge  engineering  approach  in  order  to  get  at 
the  same  time  a  description  of  the  tasks  performed  by  OROs  as  well  as  the  content  of  the 
key  mental  models  and  pictures  that  are  key  ingredients  in  the  ORO's  decision  making 
process. 

We  conjectured  that  an  important  part  of  the  data  used  by  the  ORO  are  provided  by 
operators  and/or  their  supervisors.  If  this  conjecture  is  verified,  a  cognitive  task  analysis 
coupled  with  a  knowledge  engineering  approach  should  be  carried  out  in  order  to  identify 
which  data  structures  fit  the  operators'  mental  models  and  which  part  of  these  data 
structures  are  used  to  provide  information  to  OROs.  From  these  structures  it  would  be 
possible  to  identify  which  information  will  be  provided  to  the  ORO's  SASS. 

9.3  Future  Work 

As  it  has  been  shown  in  the  present  study,  the  notion  of  situation  analysis  sheds  new  light 
on  the  process  of  decision  making  in  complex  C2  settings  by  emphasizing  the  crucial  role 
played  by  human  operators.  The  difficulty  encountered  by  researchers  and  developers  to 
build  efficient  and  complete  data  fusion  systems  with  regard  to  the  JDL  model  can  be 
explained  by  the  complexity  of  fully  automating  processes  in  which  human  judgement 
plays  a  crucial  role.  Consequently,  we  suggested  that  the  JDL  model  should  be  revised  in 
order  to  take  into  account  the  human  interventions  in  general  and  the  process  of  situation 
analysis  in  particular.  Currently,  Levels  2  and  3  of  the  revised  version  of  the  JDL  model 
(1998)  are  not  well-understood.  Hence,  on  the  theoretical  side ,  the  next  recommended 
step  is  to  further  study  the  main  works  that  have  been  performed  by  several  teams  on  JDL 
levels  2  and  3  in  order  to  identify  their  main  characteristic  elements  (concepts,  resolution 
techniques,  systems  architectures,  etc.)  and  to  relate  them  with  the  model  of  situation 
analysis  proposed  in  the  current  report.  Such  a  study  would  provide  several  outcomes: 

-  identify  the  main  problems  that  have  been  encountered  when  trying  to  automate  JDL 
levels  2  and  3: 
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-  identify  the  main  solutions  proposed  to  solve  those  problems  as  well  as  the  advantages 
and  limits  of  those  solutions; 

-  study  how  the  notion  of  situation  analysis  may  help  understand  those  problems  and 
provide  a  framework  for  integrating  already  available  solutions  and  open  directions  for 
new  solutions. 

On  the  practical  side,  the  next  recommended  step  is  to  perform  a  cognitive  task  analysis 
coupled  with  a  knowledge  engineering  approach  of  the  activities  of  an  ORO  as  well  as  of 
key  members  of  his  supporting  team.  From  the  perspective  of  the  creation  of  a  SASS,  the 
knowledge  engineering  work  should  enable  designer  to  identify  the  key  data  and 
knowledge  structures  that  will  be  used  to  create  a  situation  display  that  will  fit  the  mental 
models  of  the  S  ASS’s  users. 

With  respect  to  the  identification/representation  of  an  ORO's  mental  models/pictures,  it 
might  be  useful  to  gather  various  samples  of  the  data/information  used  by  an  ORO  when 
trying  to  gain  situation  awareness:  computer  data,  information  given  by  operators  and 
their  supervisors,  maps,  drawings,  sketches,  etc.  All  these  elements  may  contribute  to  the 
creation  and  maintenance  of  the  ORO's  mental  models  relative  to  the  attended  situation; 

Thanks  to  the  knowledge  engineering  phase,  knowledge  engineers  will  be  able  to: 

•  Develop  conceptual  model(s)  of  the  situation  from  an  ORO's  point  of  view 

•  Identify  which  data/information  are  needed  to  generate  the  situation  conceptual 
model(s) 

•  Identify  ways  to  enhance  the  situation  model  when  the  ORO  is  gaining  a  better 
understanding  of  the  situation 

We  can  conjecture  that  a  situation  model  will  not  be  completely  generated  automatically 
by  a  SASS.  Given  that  hypothesis,  it  will  be  interesting  to  characterize  how  an  ORO 
could  orient  the  SASS  in  order  to  get  more  information  in  order  to  adjust  the  situation 
model  (and  display)  so  that  it  fits  his  mental  models  and  pictures  in  a  better  way.  The 
SASS  will  need  directions  from  its  user  in  cases  in  which  the  acquired  information  is 
incomplete,  ambiguous,  noisy,  contradictory,  etc. 

Based  on  these  studies  and  models,  the  next  step  would  be  to  develop  a  base  architecture 
for  the  SASS  and  to  develop  a  prototype  in  order  to  evaluate  the  feasibility  of  the 
proposed  approach. 
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1.  Inroduction 

In  this  document,  we  give  an  overview  of  the  process  of  the  situation  assessment  (SA)  and 
threat  assessment  (TA)  in  the  context  of  the  data  fusion  process  (DF)  (Antony  1995;  Waltz 
and  Llinas  1990).  Our  first  objective  is  to  provide  a  general  picture  of  the  main  characteristics 
of  the  SA  and  TA  processes  as  they  are  usually  presented  in  the  open  literature.  Our  second 
objective  is  to  identify  in  which  areas  workable  solutions  have  been  proposed  and  in  which 
areas  satisfying  solutions  have  not  been  proposed  yet. 

The  DF  process  is  required  within  any  application  where  a  large  amount  of  data  is  to  be 
collected  and  processed  as  in  military  intelligence  gathering,  command  and  control  (C2), 
weather  prediction  applications,  etc.  In  (Hall  and  Llinas  1 997)  the  process  of  Data  Fusion  or 
Information  Fusion  is  defined  as  follows:  “The  integration  of  information  from  multiple 
sources  to  produce  specific  and  comprehensive  unified  data  about  an  entity^. 

Data  are  remotely  sensed  by  different  techniques  (e.g.  radar,  sonar,  satellite  ...).  Today,  the 
automation  of  such  a  process  is  becoming  more  and  more  crucial,  especially  in  military 
applications  for  different  reasons: 

-  The  ever-increasing  difficulty  to  identify  and  track  targets,  due  to  the  permanent 
technological  improvement; 

Situations  and  threats  are  increasingly  complex; 

-  The  distributed  character  of  sensed  data; 

-  Due  to  the  increasing  amount  of  data  to  be  processed  during  short  intervals  of  time 
(multisensor  data),  human  operators  can  be  overloaded  and  may  make  sub-optimal 
decisions  (Pedersen  et  al.  1999). 

Obviously,  it  may  be  argued  that  single-source  data  can  avoid  redundant  information  and 
therefore  reduce  the  amount  of  data  to  be  processed.  However,  redundant  data  can  be  of 
great  importance  since  they  validate  information  in  contexts  in  which  one  must  deal  with 
uncertainty  and  incompleteness. 

In  general,  the  DF  process  can  be  divided  into  two  processing  levels.  At  the  first  level,  we 
deal  with  numerical  processes  and  products  (linear  and  non-linear  estimation  techniques, 
pattern  recognition,  statistical  operations).  At  the  second  level,  the  processing  is  of  a 
symbolic  nature  (AI  techniques),  the  products  belong  to  more  abstract  levels. 

In  the  first  pan  of  the  present  report,  we  describe  three  generic  models  of  the  DF  process  that 
are  indicative  of  the  common  understanding  of  this  process  in  the  open  literature.  We  also 
briefly  mention  some  main  functions  that  are  associated  to  the  SA  and  TA  processes  and 
show  their  dependency  on  the  application  domain.  We  also  discuss  the  complexity  of  the 
problem  solving  process  in  military  context. 

In  the  second  part,  we  discuss  the  practical  solutions  that  have  been  proposed  to  manage  parts 
of  the  DF  process.  It  appears  that  few  systems  are  able  to  support  the  SA  and  TA  processes 
because  of  the  complexity  of  the  activities  involved. 

Finally,  in  the  last  (third)  part,  we  discuss  several  techniques  that  have  been  developed  in  the 
field  of  artificial  intelligence  (AI)  and  applied  to  the  DF  domain.  The  proposed  methods, 
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although  interesting,  have  various  limitations  and  do  not  lead  to  a  complete  solution  based 
solely  on  AI  techniques.  Recently,  multi-agent  techniques  have  been  proposed  to  get  various 
specialised  systems  collaborate  towards  a  solution  to  complex  problems.  Some  of  these 
multi-agent  techniques  have  been  applied  to  the  DF  problem. 

In  conclusion,  we  raise  several  issues  that  have  rarely  been  addressed  in  the  Data  Fusion 
domain  and  that  can  help  analysts  gain  a  better  understanding  of  the  SA  and  TA  processes. 


2.  PART  1 :  Theoretical  Data  Fusion  Models 

In  this  part,  we  present  different  DF  fusion  models  described  in  the  literature.  The  first  three 
sections  present  three  models  (including  SA  and  TA  processes)  from  different  points  of  view. 
Next,  we  describe  the  elements  and  functions  required  by  the  SA  and  TA  processes.  Finally, 
we  present  some  theoretical  models  that  are  used  in  military  problem-solving  to  represent  the 
main  elements  relevant  to  the  SA/TA  processes. 

2.1  A  three-level  Junctional  model  ofDF 

A  detailed  study  was  ordered  by  the  Joint  Directors  of  Laboratories  Data  Fusion  Subpanel 
(JDLDFS)  in  order  to  define  a  general  framework  for  DF  problems  (White  and  Llinas  1990) 
(Hall  and  Llinas  1997).  This  study  has  led  to  the  definition  of  two  models.  The  first  model 
has  three  levels  (see  Figure  1): 

-  Level  1 :  Estimation  of  enemy’s  identity  and  position; 

-  Level  2:  Situation  Assessment  (hostile  or  friendly):  “A  process  by  which  the 
distributions  of  fixed  and  tracked  entities  are  associated  with  environmental,  doctrinal 
and  performance  data ”  (Hall  and  Llinas  1997); 

-  Level  3:  Threat  Assessment  (in  a  hostile  situation):  “A  structured  multi-perspective 
assessment  of  the  distributions  of  fixed  and  tracked  entities  which  result  in  estimates  of 
( e.g .)  (Hall  and  Llinas  1997)  : 

-  expected  courses  of  actions; 

-  enemy  lethality; 

unit  compositions  of  deployment; 

-  functional  networks  (e.g.  supply,  comms); 
environmental  effects ”. 

It  is  important  to  mention  that  processing  becomes  more  abstract  as  we  move  to  higher  levels. 
At  Level  1,  products  result  from  single  and  multi-source  processing  that  involve: 

Tracking  (Chang  1994;  Desbois  1998;  Ding  and  Hickey  1999;  Roy  et  al.  1999;  Roy  et  al. 
1998;  Shar  and  Rong  Li  1999).  “A  process  which  generally  employs  both  correlation  and 
fusion  component  processes  to  transform  sensor  measurement  into  updated  states  and 
covariance  for  entity  tracks' ’  (Hall  and  Llinas  1997). 

Correlation.  "A  decision  making  process  which  employs  an  association  technique  as  a 
basis  for  allocating  sensor  measurements  to  the  fixed  or  tracked  location  or  entity ”  (Hall 
and  Llinas  1997). 

Alignment.  ” Processing  of  sensor  measurements  to  achieve  a  common  time  base  and 
spatial  reference ”  (Hall  and  Llinas  1997))  and  association  (“a  process  by  which  the 
closeness  of  sensor  measurements  is  completed'  (Hall  and  Llinas  1997)). 
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This  implies  sampling  the  external  environment  with  multiple  sensors  and  exploiting  other 
available  sources.  The  resulting  products  are  the  position  and  identity  estimates  of  the  targets 
or  platforms1  in  the  composite  battlefield  (Chang  and  Lu  1995;  Cowden  1995;  Jungert  1999; 
Zhang  and  Jiao  1998). 

At  Level  2,  the  fusion  process  involves  situation  abstraction  and  situation  assessment 
(Bergeron  et  al.  1998;  Campos  and  Llinas  1998;  Fracker  1988).  Situation  abstraction  consists 
of  the  construction  of  a  generalised  representation  from  incomplete  data  in  order  to  give  a 
contextual  interpretation  of  the  distribution  of  forces  determined  at  the  first  level.  The 


Figure  1.  DFS  product-oriented  model  of  the  data  fusion  process  ( White  and  al. 

1988). 

uncertainty  of  the  information  contained  in  the  modelled  situation  raises  multiple  hypotheses 
about  the  elements  characterising  this  situation  (Cohen  1984;  Cohen  1985;  Cohen  and 
Grinberg  1983;  Decker  and  Lesser  1994;  Li  et  al.  1999;  Rogova  and  Losiewicz  1999).  The 
situation  assessment  function  refines  these  hypotheses  and  selects  a  finite  set  of  important 
alternatives  for  further  analysis. 

At  Level  3,  the  fusion  process  involves  threat  and  risk  assessment  (Bergeron  et  al.  1998; 
Hayslip  and  Rosenking  1989;  Paradis  et  al.  1997),  considering  the  ability  of  friendly  forces  to 
effectively  counter  the  enemy.  This  assessment  must  also  take  into  account  the  vulnerability 


1  Platforms  are  the  raised  enemy’s  structures. 
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of  friendly  forces  during  the  confrontation.  This  level  differs  from  the  second  one  in  that  the 
results  should  provide  a  true  quantification  of  the  enemy’s  capacities  to  threaten  friendly 
forces.  It  should  also  give  estimates  about  the  enemy’s  intentions.  At  the  second  level,  the 
results  are  limited  to  the  identification  of  the  enemy’s  behavioural  patterns,  without 
interpreting  them  nor  determining  their  consequences. 

2.2  An  architectural  model  of  DF 

The  second  model  proposed  in  (White  and  Llinas  1990)  is  viewed  from  a  system  architecture 
point  of  view.  The  main  characteristic  of  this  model  is  its  distributed  nature  (see  Figure  2). 
Each  node  corresponds  to  a  set  of  processing  units. 

The  bus-type  elements  support  the  notion  of  connectivity.  As  we  can  see  in  Figure  2,  there 
are  two  types  of  connectivity:  interconnectivity  and  intraconnectivity.  The  first  one  connects 
different  processing  units  with  the  external  environment.  The  second  one  connects  the 
processing  units  of  a  node.  These  processing  units  provide  the  same  functions  as  those  in  the 
first  model  described  in  Section  2.1:  similar  source  integration,  dissimilar  source  integration, 
situation  abstraction,  situation  assessment  and  threat  assessment. 


Figure  2.  DFS  system-architecture-oriented  model  of  the  data  fusion  process . 


At  the  first  level,  a  distinction  is  made  between  single  and  multi-source  data  fusion.  The  final 
processing  product  is  called  the  “ Dynamic  and  Integrated  Situation  Representation” .  The 
relationship  between  this  product  and  the  different  processing  units  is  the  same  as  the 
relationship  that  can  be  found  between  knowledge  sources  and  a  blackboard  in  knowledge- 
based  systems. 

A  dynamic  representation  of  the  assessed  situation  and  the  assessed  threat  is  the  key  element 
for  the  decision-making  process  (Paradis  et  al.  1999).  In  fact,  decisions  are  always  made  with 
regard  to  new  incoming  events.  The  interpretation  of  these  events  can  be  viewed  as  a  change 
in  the  current  situation  state  and  the  related  current  TA. 
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2.3  A  four-level  functional  model 

The  functional  model  of  DF  proposed  by  (White  et  al.  1993)  is  divided  into  four  levels  as 
shown  in  Figure  3: 

Level  1 :  this  level  is  characterised  by  four  functions: 

Detection  of  objects; 

-  Association  of  the  detected  objects  with  other  identified  objects; 

Refinement  of  objects'  attributes:  position,  velocity,  etc.; 

Classification  and  identification  of  detected  objects:  “a  process  by  which  some  level  of 
identity  of  an  entity  is  established,  either  as  a  member  of  a  class,  a  type  within  a  class,  or  a 
specific  unit  within  a  type"  (Hall  and  Llinas  1997). 

-  Level  2  (situation  refinement):  this  level  is  an  extension  of  level  1  which  aims  at 
enhancing  the  situation  description  produced  at  that  level,  in  terms  of  completion, 
consistency,  and  abstraction.  It  comprises  three  functions: 

-  Error  correction; 

Ambiguity  resolution  of  the  results  provided  by  Level  1 . 

High-level  interpretation  of  all  information  coming  from  the  different  sensors. 

Level  3  (threat  refinement):  this  level  identifies  the  potential  intentions  of  the  enemy  and 
estimates  vulnerability  of  friendly  forces.  The  dynamic  nature  of  a  situation  implies  that 
TA  must  predict  all  possible  short-term  and  long-term  implications  of  that  situation. 

Level  4  (process  refinement):  this  level  is  in  charge  of  the  optimisation  of  the  system’s 
performances  with  regard  to  certain  objectives.  For  example,  sensors  may  collect  relevant 
but  redundant  information  if  not  controlled.  Processing  a  large  amount  of  redundant 
information  can  seriously  affect  the  performance  of  the  whole  system.  This  level  aims  at 
controlling  the  quality  of  fused  data.  It  also  ensures  the  control  of  the  three  other  levels  in 
order  to  orient  and  focus  reasoning  activities  at  each  level. 

The  first  stage  of  Level  2  is  an  abstraction  of  the  situation  which  can  be  a  generalisation  or  a 
specialisation  of  the  situation.  Generalisation  aggregates  or  abstracts  information  in  order  to 
provide  a  situation  awareness  with  respect  to  higher-level  entities  such  as  activities.  For 
example,  a  com  field  will  be  associated  to  fanning  activity.  Specialisation,  on  the  other  hand, 
results  from  a  top-down  reasoning  process  that  aims  at  deducing  or  inferring  subordinate 
elements  or  entities.  For  example,  a  farming  activity  in  a  region  suggests  the  existence  of 
tractors.  Generalisation  and  specialisation  allow  the  definition  of  structural,  organisational, 
and  functional  relationships  among  the  different  elements  of  a  field.  Situation  abstraction 
attempts  to  provide  a  complete  representation  of  the  situation.  The  second  stage  of  Level  2  is 
the  SA  that  gives  a  dynamic  representation  of  the  situation  in  terms  of  links  between  the 
observed  events  and  the  states  of  entities  and  organisations. 

A  system  architecture  based  on  the  four-level  model  described  above  may  be  centralised  or 
distributed.  Distributed  DF  systems  vary  considerably  with  respect  to  the  actual  distribution 
of  data,  processing  and  control.  The  distribution  of  each  of  these  three  elements  allows 
concurrent  processing  in  the  system.  Centralised  systems  have  the  advantage  of  accessing 
data  that  are  physically  stored  in  the  same  place. 

In  a  distributed  context,  data  are  located  in  different  places.  So,  information  may  be  lost 
because  of  local  processing  and  communication.  A  detailed  description  of  distributed  systems 
can  be  found  in  (Manyika  1994).  The  main  drawback  of  centralised  systems  is  the  possibility 
of  a  general  breakdown  of  the  system  because  of  the  failure  of  a  major  component. 
Blackboard-based  architectures  are  widely  used  for  the  design  of  distributed  systems 
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(Jagannathan  et  al.  1989).  These  architectures  allow  a  centralised  control  and  communication 
between  spatially  distributed  knowledge-based  systems. 


Data  fusion  domain 


Figure  3  :  High-level  Junctional  model  of  the  data  fusion  process  (Antony 
1995). 


2.4  The  Joint  Directors  of  Laboratories  Model  Revision 

The  model  described  in  section  2.3  is  widely  used  to  categorise  data  fusion  functions. 
However,  data  correlation  and  combination  and  inference  are  also  needed  in  a  wide  range  of 
non-military  applications.  In  non-military  applications,  names  such  as  threat  assessment  or 
threat  refinement  should  be  changed.  Furthermore,  there  is  no  standard  interpretation  of  the 
JDL  model’s  levels.  They  have  variously  been  used  as  referring  to  the  kinds  of  association  or 
estimation  processing  involved,  the  kinds  of  entities  being  characterised  and  the  degree  to 
which  the  data  used  in  the  characterisation  have  already  been  processed.  Finally,  separating 
the  functions  of  data  fusion  into  levels  has  led  several  persons  to  associate  an  implicit  order  to 
them:  first  do  level  1 ,  then  level  2,3  and  finally  level  4. 

The  revised  version  of  the  JDL  model  aims  at  redefining  the  levels  in  order  to  solve  such 
problems.  The  new  version  of  the  model  has  five  levels  (Steinberg  et  al.  1998): 

Level  0  -  Sub-Object  Data  Assessment:  estimation  and  prediction  of  signal/object 
observable  states.  This  level  involves  hypothesising  the  presence  of  a  signal  and 
estimating  its  state.  Level  0  processing  includes  signal  detection  from  a  time-series  of 
data'and  feature  extraction  from  images. 

Level  1  -  Object  Assessment:  this  level  focuses  on  objects  aiming  at  estimating  and 
predicting  their  states  on  the  basis  of  observation-to-track  association,  continuous  state 
estimation  and  discrete  state  estimation. 
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-  Level  2  -  Situation  Assessment:  estimation  and  prediction  of  relations  among  entities. 
The  relations  can  be  represented  as  a  network.  This  network  allows  associating  tracks  and 
other  information  into  aggregations.  Any  type  of  relation  can  be  considered  according  to 
the  mission  or  application:  it  might  be  physical,  organisational,  functional,  informational, 
perceptual,  etc. 

-  Level  3  -  Impact  Assessment:  estimation  of  the  impact  of  the  assessed  situation.  This 
level  takes  into  account  the  impact  of  the  planned  course  of  actions  for  each  participant, 
so  that  it  is  possible  to  determine  interaction  of  an  action  with  the  other  actions,  and  its 
interaction  with  the  environment. 

Level  4  -  Process  Refinement:  it  involves  planning  and  control  rather  than  estimating. 
This  level  essentially  manages  the  assignment  of  tasks  to  resources. 

In  conclusion,  the  main  functions  of  the  first  two  levels  are  detection  and  estimation  of 
signals  or  objects.  The  next  two  levels  aggregate  these  objects  into  situations  thanks  to  the 
identification  of  relations  between  objects.  Finally,  the  last  level  deals  with  control  aspects 
such  as  planning  actions  with  respect  to  available  resources. 

2.5  Functional  requirements  for  SA  and  TA 

In  the  last  section,  we  presented  a  general  overview  of  a  functional  architecture  for  the  DF 
process.  The  defmition  of  a  class  of  architectures  for  SA  and  TA  is  a  more  difficult  task 
because  these  processes  are  relatively  ill-defined  (see  (Hall  and  Garga  1 999)  for  a  description 
of  the  pitfalls  in  Data  Fusion  and  how  to  avoid  them).  In  this  section,  we  identify  the  main 
characteristics  that  SA  and  TA  processes  are  expected  to  have,  by  defining  the  elements  and 
functions  that  they  should  include  (Waltz  and  Llinas  1990,  p.28-33). 

2.5.1  Elements  and  functions  of  the  SA  process 

Before  abstracting  or  assessing  a  situation  using  data  acquired  from  Level  1,  countermeasure 
activities  must  be  assessed,  for  example,  the  force  disposition/location/deployment  in  an 
environmental  or  socio-political  context  or  both  (see  Figure  4).  Such  an  organisational 
description  leads  to  the  definition  of  a  set  of  functions  required  in  a  SA  process.  Figure  5 
presents  an  overview  of  the  relevant  functions  for  the  SA  process  in  the  environmental  or 
socio-political  context. 

Ideally,  the  SA  system  must  yield  results  that  a)  reflect  the  true  situation  and  b)  provide  a 
basis  for  event-activity  prediction  and  thereby  a  basis  for  optimal  sensor  management.  The 
SA  process  is  therefore  concerned  with  what  is  happening  and  which  events  or  activities  are 
going  to  happen.  It  focuses  on  the  behavioural  aspects  of  the  relevant  elements  contained  in 
the  area  of  interest. 

The  highest  level,  represents  the  “ struggle  of  motives  assessment  function"  (Figure  5).  This 
function  means  that  each  collection  of  data  in  a  SA  process  may  have  different  interpretations 
and  implications  in  the  decision-making  process.  Note  that  ideally,  the  SA  analyst  should  be 
applying  the  concept  of  shifting  perspectives  in  order  to  develop  an  optimal  viewpoint  of  the 
situation.  He  or  she  has  to  examine  data  from  different  points  of  view. 
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These  functions  may  be  refined  at  several  levels  of  detail.  For  example,  a  fighter  pilot's 
estimation  of  a  situation  is  not  the  same  as  that  of  a  commander  in  land  battle  in  terms  of 
time-criticality  and  complexity. 


Situation  Assessment  Elements 


Contermeasures  Elements 


Force  Disposition/  Environmental  Elements  ’  '  Socio-political 

Deployment/Location  Elements 


—  Radiation  j—  Econimic 

—  Terrain  * —  Political 

-Hyd6- 

' .  Intangibles  - - — - -  Meteorol|ical 


-Morale 

-Psychological  state 

—  Level  of  training 
-National  characteristics 

-  Stability  under  stress 
-Stregth  of  will 


,  Friendlv 


■ .  Adjacent 


Figure  4.  Elements  of  the  SA 

In  general,  SA  is  clouded  with  uncertainty  because  of  the  difficulty  to  estimate  the  enemy’s 
intent.  For  this  reason,  different  situation-estimating  countermeasures  must  be  taken  by 
friendly  commanders.  These  countermeasures  are  called  CC&D  techniques: 

Concealment:  methods  to  prevent  sensors  from  observing  deployments,  capabilities, 
intentions. 

Cover:  methods  (e.g.  camouflage  and  avoidance)  to  deny  the  adversary  the  intelligence 
data  needed  for  carrying  out  its  operations. 

Deception:  injection  of  false  or  misleading  data. 

Ambiguity:  Generating  situations  that  have  multiple  interpretations  (especially 
concerning  intent). 

As  we  can  observe  in  this  brief  overview,  the  types  of  SA  functions  proposed  in  (Waltz  and 
Llinas  1990,  p.28-33)  are  very  much  dependent  on  the  kind  of  application  domain  that  is 
considered.  For  example,  the  functions  presented  in  Figure  5  will  not  apply  to  the  tasks  of  the 
Operations  Room  Officer  (ORO)  in  the  HALIFAX  class  frigates. 
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Situation  Assessment  Functions 


Figure  5.  Situation  Assessment  Functions 
2.5.2  Elements  and  functions  of  the  TA  process 

For  each  situation,  the  SA  process  must  be  followed  by  a  TA  process.  This  process  qualifies 
the  situation  quantitatively  as  much  as  possible,  in  order  to  estimate  the  enemy’s  effective 
capabilities,  its  intentions,  and  the  scope  of  the  threat.  There  are  some  similarities  between 
the  elements  of  the  SA  and  TA  processes,  such  as  the  counter-measure  activities.  Figure  6 
shows  the  elements  of  TA  according  to  (Waltz  and  Llinas  1990,  p.28-33). 

TA  is  a  process  where  different  elements  are  quantified  in  order  to  determine  the  enemy’s 
capacities,  vulnerability  and  intent: 

Strength.  Enumeration  of  the  number  and  size  of  the  enemy  units. 

Composition.  Structure  of  enemy  forces,  organisation  and  weapons. 

Location  and  disposition.  Description  of  the  geographical  location  of  the  enemy, 
including  the  fire  support  elements,  command  and  control  facilities,  air,  naval,  missile 
forces. 

Availability  of  reinforcement.  Description  of  the  enemy  reinforcement  capabilities  in 
terms  of  ground,  air,  naval,  missile  . . . ;  communication  and  transportation  means. 

-  Movements  and  activities.  Description  of  the  latest  known  enemy  activities  in  the  area. 

-  Logistics.  Description  of  levels  of  supply  ability:  capacity  of  beaches,  ports,  airports  . . . 
Operational  capability  to  launch  missiles.  Relevant  factors:  characteristics  of  missile 
systems,  launch  rates,  size  and  locations  of  stockpiles  . . . 

Serviceability  and  operational  rates  of  aircraft.  A  total  aircraft  inventory  by  type, 
performance  and  characteristics  of  operational  aircraft,  . . . 


Al-12 


-  Operational  capabilities  of  combatant  vessels.  Description  of  the  number,  type  and 
operational  characteristics  of  ships,  boats, . . . 

Technical  characteristics  of  equipment.  Description  of  the  technical  characteristics  of 
major  items  of  equipment  in  the  enemy  inventory. 

-  Electronics  intelligence.  Intelligence-gathering  capability  using  electronics  devices. 

-  Nuclear  and  CB  weapons.  Types  and  characteristics,  delivery  capabilities,  employment 
policies  and  techniques  . . . 

Significant  strengths  and  weaknesses.  An  estimate  of  the  significant  enemy  strengths  and 
weaknesses  can  be  developed  from  the  facts  presented  in  the  preceding  list. 

As  for  the  SA,  TA  has  also  a  set  of  required  functions  as  shown  in  Figure  7.  Although  more 
generic  than  the  SA  functions,  we  can  observe  that  the  TA  functions  proposed  in  (Waltz  and 
Llinas  1990,  p.28-33)  need  to  be  adapted  to  each  application  domain. 


Threat  Assessment  Elements 


Figure  6.  The  elements  ofTA. 


2.6  The  military  problem-solving  process 

As  we  stated  before,  the  situation  and  threat  assessment  processes  are  based  on  multiple- 
perspectives  and  contexts.  In  a  real  context,  we  can  have  a  situation,  where  two  commanders 
have  two  opposite  viewpoints  on  the  problem  to  be  solved,  because  they  follow  two  different 
theories  or  have  different  understandings  of  the  situations.  A  well-defined  problem-solving 
process  may  help  these  commanders  to  concentrate  on  the  enemy  rather  than  to  try  to  agree 
on  the  situation. 
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Figure  7.  Functions  ofTA. 

Several  researchers  (Reitman  1965)  have  described  this  problem  in  an  abstract  way  by  using 
a  state  transition  model.  There  are  many  types  of  state-transitions  as  shown  in  Figure  8  from 
(Dieterly  1980).  Nodes  correspond  to  states  and  links  correspond  to  transitions  from  an  initial 
state  to  a  final  state.  Dashed- line  nodes  and  links  respectively  correspond  to  unknown  states 
and  unknown  transitions. 

Each  of  these  situations  leads  to  different  difficulties.  Furthermore,  there  are  situations  where 
one  action  may  lead  to  different  states,  or  multiple  actions  may  lead  to  the  same  end  state  (see 
Figure  9).  The  combination  of  cases  presented  in  Figures  8  and  9  leads  to  40  possible 
problem  situations.  In  (Reidelhuber  1984)  a  model  of  tactical  decision  making  is  presented. 
This  hierarchical  model  integrates  various  Command,  Control  and  Communication  (C3) 
functions  with  battlefield  state  estimates  and  can  be  considered  as  an  elaboration  of  the 
concepts  given  in  Figures  8  and  9. 

I  II  III  IV  V  VI  VII  VIII 


Legend 

I:  all  problem  aspects  unknown 
II:  end  state  known 
III:  approach  known/fixed 
IV:  approach,  end  state  known 


V:  initial  state  known 
VI:  initial  state,  approach  known 
VII:  initial,  end  states  known 
VIII:  all  problem  aspects  known 


Figure  8.  Decision-problem  model. 
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Another  relevant  decision-making  model  is  the  SHOR  (Stimulus-Hypothesis-Option- 
Response)  model  (Whol  and  al.  1984)  which  aims  at  dealing  explicitly  with  both  information 
input  uncertainty  and  consequence-of-action  uncertainty  in  military  problem  solving. 

Let  us  now  look  at  particular  applications  in  order  to  see  how  data  fusion  is  practically  put 
into  work. 


Single  Multiple  Multiple  Multiple  Multiple 

End  States  Transitions  End  States 

and 

Transitions 

Figure  9.  Dieterly 's  classes  of  conditions. 


3.  PART  2:  Practical  Solutions 

In  this  part,  we  briefly  review  some  real-world  architectures  where  a  DF  process  takes  place. 
We  will  describe  the  required  functions  of  such  architectures.  We  will  also  discuss  some 
implementation  issues  related  to  the  use  of  knowledge-based  and  expert  systems  techniques. 

3.1  Real-world  Architectures 

In  general,  in  practical  systems,  the  DF  process  only  represents  a  small  portion  of  the  overall 
system  (15-20  %).  In  (Ballard  and  Rippy  1994),  a  system  for  performing  situation  assessment 
in  Next-Generation  Army  Helicopters  is  developed  which  is  composed  of  three  subsystems: 
recognition,  assessment  and  prediction.  In  (Bass  2000),  the  author  discusses  the  cyberspace 
intrusion  detection  issue,  and  its  relation  to  the  data  fusion  process.  With  intrusion  detection 
systems,  designers  face  the  challenge  of  combining  data  from  different  heterogeneous  sources 
into  a  coherent  “picture”  which  aims  at  improving  the  process  of  evaluation  of  the  cyberspace 
security.  It  is  important  to  notice  that  in  cyberspace  intrusion  detection  systems,  sensors  are 
different  because  the  environmental  dimension  is  different.  Instead  of  trying  to  detect  a 
missile  launch  and  its  supersonic  flight  in  the  atmosphere,  cyberspace  sensors  observe 
information  flowing  through  electronic  networks.  The  decision  system  for  cyberspace 
intrusion  detection  is  based  on  a  so-called  OODA  model  (Observe,  Orient,  Decide  and  Act) 
which  is  widely  used  in  military  information  operations.  The  decision  support  process  for 
situational  awareness  (Chalmers  1997;  Drake  and  Atwell  1997)  is  tightly  coupled  with  the 
DF  process.  This  cyber-fusion  process  requires  the  use  of  different  techniques  ranging  from 
processing  algorithms  and  statistical  estimations  to  heuristic  methods  such  as  expert  systems, 
to  assess  situations  and  threats  in  cyberspace. 

These  examples  show  that  in  a  functional  architecture,  the  DF  process  is  only  a  small  portion 
of  the  system.  Such  an  architecture  would  include  the  following  functions  (Waltz  and  Llinas 
1990,  p.  23): 

Communication :  the  system  is  never  isolated  from  the  rest  of  the  environment.  Entities 

located  in  the  environment  must  be  able  to  communicate  with  the  system  so  that  it  can 
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update  its  representation  of  the  environment  and  of  the  current  situation  as  well  as  the 
threat  related  to  this  situation  (Greeway  et  al.  1996).  In  addition,  human  operators  must  be 
able  to  communicate  with  the  system,  either  to  interpret  information  or  to  update 
information  resulting  from  the  decision  making  process. 

Data-base  management:  data  captured  from  the  external  environment  may  be  voluminous 
and  of  different  natures.  These  data  give  way  to  several  interpretations  because  of  their 
incomplete  nature.  Hence,  we  may  have  to  manage  several  potential  ongoing  scenarios  as 
combinatorial  problems. 

-  Person-Machine  Interface :  Continuous  changes  observed  in  the  external  environment 
lead  to  continuous  interactions  between  human  operators  and  the  system.  Very  often, 
these  interactions  require  time-constrained  responses  that  vary  according  to  the  type  of 
application  in  which  the  DF  process  takes  place.  In  order  to  ensure  good  quality 
interactions,  the  system  interface  must  be  carefully  designed. 

This  list  of  components  is  not  exhaustive.  A  system  may  have  other  functions  depending  on 
the  application  needs.  From  a  computational  point  of  view,  the  DF  cannot  be  separated  from 
the  other  components  of  the  system.  Consequently,  xhe  DF  system  can  be  constrained  by 
external  events  causing  sub-optimal  performances.  For  example,  if  the  sensors  are 
momentarily  inaccessible,  the  DF  process  may  be  blocked,  thus  preventing  the  system  from 
obtaining  optimal  results.  The  interdependencies  between  the  system's  components  require  in 
some  cases,  the  use  of  parallel  processing  techniques  at  the  first  level  in  order  to  avoid  system 
blockage.  These  techniques  may  also  be  required  in  a  time-constrained-response  context, 
such  as  in  the  military  aviation  domain  (Pipe  1992),  where  pilots  can  delegate  control  to  the 
computer  onboard  the  aircraft.  They  can  thus  preserve  “ situation  awareness ”  when  events 
change  faster  than  human  recognition. 

3.2  Implemented  Systems 

In  this  section,  we  only  describe  knowledge-based  and  expert  systems  techniques.  Generally, 
the  goal  of  such  systems  is  to  produce  an  estimate  of  the  hostile  force's  structure  or  order  of 
battle.  The  Expert  System  for  Intelligence  Analysis  Support  (ESIAS)  project  proposes  one  of 
the  most  comprehensive  systems,  at  least  in  its  conceptual  treatment  of  the  SA  problem.  The 
work  reported  in  (King  and  al.  1986;  Ruoff  1988)  emphasises  three  aspects  of  this  system:  the 
development  of  ESIAS  and  the  Situation  Assessment  knowledge-based  system  (KBS),  the 
explanation  facility  of  ESIAS  and  a  system  called  DECision  Aid  Development  and 
Evaluation  (DECADE),  which  is  a  development  environment  for  database-system-based  aids 
for  situation  assessment.  The  goal  of  this  work  was  to  develop  an  environment  to  build 
efficient  aids  that  evaluate  threat  capabilities  and  actions,  infer  threat  intentions  and  predict 
undetected  and  future  situations.  The  authors  used  a  technique  called  “ conceptual  knowledge 
modelling ”  to  develop  the  Command,  Control,  Communication  and  Intelligence  (C3I) 
conceptual  knowledge  model  which  is  composed  of  two  components: 

-  The  C3I  conceptual  structure  model  (CS  model), 

-  The  C3I  situation  assessment  behaviour  model  (SAB  model). 

The  SAB  model  presents  both  the  functional  and  the  procedural  views  of  the  SA  process.  The 
CS  model  is  used  to  define  requirements,  whereas  the  SAB  model  provides  a  high-level, 
implementation-independent  model  of  the  procedural  knowledge  to  be  represented  in  the 
knowledge  base.  These  views  are  then  extended  to  provide  a  detailed  view  of  the  SA  process 
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also  point  out  that  much  of  the  research  and  development  works  on  the  data  fusion  process 
have  been  concentrated  on  Level  1. 


4.  PART  3:  AI  Technology 

In  this  part,  we  discuss  the  benefits  and  drawbacks  of  Al-based  technologies  in  the  DF 
process.  We  review  multi-agent  techniques  that  are  useful  for  the  DF  process.  We  end  this 
part  with  a  discussion  about  the  human  factors  in  the  DF  process  and  a  conclusion. 

4.1  Role  of  Artificial  Intelligence  in  the  DF  process 

Artificial  intelligence  techniques  are  widely  used  in  many  application  domains  where 
different  aspects  are  to  be  addressed  at  once  and  where  the  use  of  mathematical  methods 
seem  to  be  insufficient,  or  even  impossible.  Data  fusion  is  such  a  domain.  The  main  AI 
applications  used  to  solve  DF  problems  are:  expert  systems  or  knowledge-based  systems 
(KBS),  natural  language  processing  (NLP),  planning  or  plan  recognition,  learning,  and 
intelligent  assistance.  Other  techniques  used  in  the  AI  domain  are  also  worth  mentioning, 
such  as  pattern  matching  techniques  which  aim  at  searching  for  the  best  or  an  acceptable 
solution  for  a  problem  in  a  context  where  several  solutions  are  possible. 

Data  Fusion  levels  2  and  3  processing  involve  combinatorial  complex  problems  that  are 
difficult  to  tackle  even  with  modern-day  computers.  For  this  reason,  KB  approaches  can  lead 
to  solutions  that  are  satisfying  but  knowingly  sub-optimal.  Sometimes,  AI  techniques  axe 
combined  with  numerical  techniques  in  DF  experiments  (Level  1),  because  reasoning 
strategies  typically  depend  on  the  quantitative  values  of  various  parameters.  In  (Huang  and 
Lodaya  1990)  the  authors  investigate  the  performance  of  an  automated  decision-making  aid 
based  on  expert  system  technology.  This  system  which  is  used  to  assist  Airborne  Early 
Warning  (AEW)  operators  during  the  threat  assessment  task,  combines  AI  techniques  and 
numerical  algorithms. 

4.1.1  Main  techniques  provided  by  AI  technology 

The  following  AI  techniques  may  be  applied  to  the  DF  process  (Waltz  and  Llinas  1990,  p. 
427): 

•  Expert  systems  and  knowledge-based  systems  (KBS)  may  be  used  to  recover  deficiencies 
of  Level  1  algorithms  (Cowden  1995;  Shahbazian  et  al.  1998).  The  KBS  may  apply 
contextual  knowledge  and  knowledge  about  algorithm  performance  to  select  and  invoke 
the  best  algorithm  for  the  current  problem.  Knowledge  may  also  be  combined  with 
numerical  methods,  in  order  to  improve  their  adaptability.  For  Level  2  and  Level  3,  KBS 
techniques  provide  a  broad  range  of  reasoning  and  inference  strategies  to  overcome  the 
combinatorial  aspect  of  the  problems  (Bergeron  et  al.  1998;  Chaudhri  and  DesJardins 
1999;  Dorfel  and  Distelmaier  1997;  Hayslip  and  Rosenking  1989:  Ranze  1999;  Ruoff 
1988;  Yang  and  Sun  1999).  They  also  allow  a  multi-level  inferring  strategy  in  SA  and 
TA.  This  gives  the  possibility  of  top-down  and  bottom-up  reasoning  in  the  hierarchy  of 
hypothesised  threats  (potential  threats).  Further,  situation  and  threat  estimating 
performance  can  be  improved  in  distributed  or  headquarters-type  fusion  centres  by 
applying  decision  and  analysis  support  with  communicating  or  co-operating  KBSs.  The 
main  advantage  of  knowledge-based  systems  is  their  use  of  human  knowledge  to  solve 
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problems  (Cooke  1994).  They  represent  a  means  to  emulate  the  reasoning  and  inference 
processes  of  human  experts.  Usually,  solutions  are  based  on  heuristics  which  capture  the 
experts’  know-how  and  rules  of  thumb.  However,  these  heuristics  are  always  presented  as 
knowledge  fragments.  Hence,  the  problem  is  to  find  a  way  to  organise  the  decision¬ 
making  processes  using  these  fragments  in  a  useful  way.  The  most  frequently  used  KBSs 
are  rule-based  systems  (RBS).  Rule-based  systems  have  limitations  that  have  been 
addressed  by  the  AI  community.  Enhanced  techniques  have  been  proposed  for  so-called 
second-generation  expert  systems. 

•  Natural  language  processing  methods  can  be  powerful  fusion  support  tools  for  message- 
based  systems.  They  can  help  human  operators  when  analysing  and  filtering  messages 
and  textual  alarms  produced  by  the  system,  especially,  in  a  context  where  responses  to 
such  messages  are  time-constrained. 

•  Military  operations  are  always  guided  by  a  plan  or  a  set  of  plans.  These  operations  are 
complex,  involve  multiple  resources  and  goals  and  require  significant  coordination. 
Classical  plan  recognition  approaches  are  based  on  a  predictability  assumption.  This 
means  that  they  assume  that  the  planner’s  model  of  the  world  (representation  of  the 
environment)  as  well  as  the  effect  of  an  action  in  that  world  are  complete.  This 
assumption  does  not  hold  in  a  military  DF  context.  The  planner  may  lack  the 
effectiveness  required  to  achieve  a  goal.  There  may  be  no  sequence  of  primitive  actions 
that  achieve  the  goal.  In  such  a  context,  classical  planning  models  are  usually 
unsuccessful.  The  condition  for  successful  planning  is  called  the  effectiveness  condition. 
In  a  military  context,  we  may  have  either  a  predictability  assumption  or  an  effectiveness 
condition  violation  or  both.  The  problem  of  how  to  plan,  when  at  least  one  of  these 
conditions  is  violated,  is  referred  to  as  the  reactive  planning  problem.  In  this  case,  the 
system  must  take  into  account  constraints  on  the  time  left  for  re-planning.  Hayslip  and 
Rosenking  (Hayslip  and  Rosenking  1989)  proposed  a  system  architecture  composed  of 
several  planners  concurrently  operating  at  different  time  scales.  The  system  is  composed 
of  five  co-operating  expert  systems  that  communicate  through  a  blackboard.  This  work 
was  part  of  the  REACT  (Grumman's  Rapid  Expert  Assessment  to  Counter  Threats) 
system  designed  to  aid  pilots  in  air  combat  to  make  decisions  at  low  altitude  and  hilly 
terrain.  The  plan  recognition  methods  are  primarily  applied  at  Levels  2  and  3. 

•  At  Levels  2  and  3,  estimating  or  predicting  future  behaviour  is  of  main  importance 
(Ballard  and  Rippy  1994).  However,  the  ability  to  estimate  such  behaviours  is  limited 
because  of  the  complex  nature  of  processes  and  behaviours  that  can  be  encountered  in 
combat  or  even  in  peace  time.  Ideally,  more  robust  fusion  systems  would  be  able  to  adapt 
to  extreme  behaviours.  Machine  learning  techniques  could  also  be  applied  to  problems  at 
Level  1. 

•  Intelligent  assistance  aims  at  supporting  human  operators  when  achieving  various  tasks. 
In  DF  systems,  these  aids  can  be  thought  of  as  alerting  functions,  often  called  indication 
and  warning  functions  (I&W).  Modem  human-computer  interface  design  techniques 
perform  similar  attention-focusing  functions  (Achille  et  al.  1997;  Bachelder  and  Hansman 
1997;  Dorfel  and  Distelmaier  1997;  Fechtig  et  al.  1997;  Hoffmann  et  al.  1998;  Huijsing 
and  Meijer  1997;  Kuperman  et  al.  1999;  Osgood  and  Adams  1997;  Rhodenizer  et  al. 
1997).  Methods  in  this  area  differ  from  others  in  that  special  features  for  interacting  with 
humans  and  for  improving  human  performance  are  included.  Such  methods  should  be 
based  on  the  creation  of  cognitive  models.  For  example,  such  systems  can  maintain  a 
model  of  the  profiles  of  user  groups  so  that  the  system  can  adjust  to  each  user’s  problem¬ 
solving  approach.  The  only  drawback  of  intelligent  assistance  is  that  it  is  difficult  to  apply 
it  in  a  time-critical  context. 
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•  Pattern  matching  techniques  are  widely  applicable  to  DF  problems,  especially  in  the 
context  of  Level  1  object  identification  strategies.  Neural  networks  are  also  used  for 
tracking  problems  (Kuh  1998;  Whittington  et  al.  1993)  which  is  an  application  of  sensor 
data  fusion.  Pattern  recognition  methods  are  also  used  at  Levels  2  and  3  (Freeman  et  al. 
1997;  Rogova  and  Menon  1998)  as  well  as  fuzzy  logic  techniques  (Niekerk  et  al.  1999; 
Shastri  and  Wendelken  1999;  Yang  and  Sun  1999). 

4.1.2  Limitations  of  Al  techniques 

Although  many  Al  techniques  are  useful  in  the  DF  process,  many  implementation  issues  must 
be  dealt  with,  given  the  complexity  of  tasks  at  Levels  2  and  3.  Whatever  the  technique  used 
in  a  computerised  system,  the  design  of  such  a  system  depends  on  the  size  of  the  problem,  on 
the  data  error  characteristics  and  on  the  reliability /uncertainty  of  the  applied  knowledge. 

Other  DF-specific  factors  that  influence  Al-based  solutions  are  (Waltz  and  Llinas  1990,  p. 
442): 

•  Real-time  processing  requirements.  For  each  level  of  DF  process,  we  must  select  the  right 
time  granularity  for  the  system.  The  shortest  response  time  at  each  level  determines  the 
most  time-critical  part  of  the  process.  The  FUTURE  system  (De  Jongh  et  al.  1994)  is  an 
example  of  a  real-time  knowledge-based  system  that  was  used  to  help  African  Navy 
identify  threatening  events  early. 

•  Time-varying  data  and  solutions  (temporal  variance).  In  (Bonissone  and  Aragones  )  the 
authors  present  a  system  that  was  used  in  a  naval  scenario  for  ship  identification  and  in  a 
tactical  aerial  situation,  providing  the  pilot  with  information  about  the  intent  of  potential 
threats. 

•  Combined  symbolic  and  numerical  functions.  Each  of  these  functions  requires  a  specific 
programming  language.  That  raises  the  issue  of  inter-language  functionality.  When  it  is 
impossible  for  a  language  to  call  another,  each  process  must  be  executed  separately,  and 
the  data  and  results  have  to  be  communicated  through  a  shared-memory  base. 

•  Large  data  /  knowledge  requirements. 

•  Significant  levels  of  uncertainty  in  data  and  knowledge. 

•  Human  operator  foibles  and  idiosyncrasies. 

Al  techniques  have  made  good  progress  during  the  past  ten  years.  They  often  provide 
interesting  solutions  to  difficult  and  well-defined  problems.  However,  in  a  DF  process,  many 
different  techniques  may  be  required  to  tackle  the  variety  of  DF  activities.  Various  sub¬ 
systems  need  to  collaborate  in  order  to  build  a  solution.  Blackboard  architectures  have  been 
proposed  in  the  eighties  to  make  several  processes  (called  knowledge  sources)  co-operatively 
build  solutions  to  complex  problems.  In  such  an  architecture,  the  shared  memory  (the 
blackboard)  appears  as  a  bottleneck  that  may  impair  the  whole  system’s  performance.  This 
problem  can  be  handled  using  multi-agent  systems. 

4.2  Multi-agent  techniques 

A  detailed  definition  of  a  multi-agent  system  is  out  of  the  scope  of  this  report.  Simply  put,  a 
multi-agent  system  is  composed  of  several  agents  that  must  co-operate  via  communication  or 
other  modes  of  interaction  in  order  to  solve  a  common  problem.  Each  agent  participates  in  the 
resolution  of  the  problem.  The  basic  assumption  is  that  the  sum  of  the  agents’  individual 
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capabilities  and  their  interaction  ability,  will  bring  better  solutions,  than  merely  the  sum  of 
their  individual  capabilities  (each  agent  working  alone). 

Such  techniques  are  especially  well-suited  for  applications  which  include  heterogeneous  and 
distributed  aspects.  In  general,  in  such  applications,  data  and  processing  control  are 
decentralised,  and  each  agent  will  have  a  set  of  responsibilities  associated  with  a  domain  in 
the  environment  where  agents  are  supposed  to  evolve.  Decentralisation  can  be  physical 
(agents  are  at  different  locations)  or  functional  (e.g.  hierarchical  design  of  a  system). 

This  is  the  case  of  the  DF  domain  applications  (Bastos-Filho  et  al.  1998;  De  Jongh  et  al. 
1994;  Zhongyan  et  al.  1999).  Hence,  multi-agent  techniques  seem  to  be  well-adapted  for 
distributed  problem  solving  within  the  DF  process  (Linderman  1999).  In  (Carver  et  al.  1993; 
Carver  et  al.  1995),  the  authors  address  the  problem  of  how  a  set  of  co-operating  agents  must 
operate  in  order  to  converge  towards  an  acceptable  solution  with  a  minimum  amount  of  time 
and  communication,  and  what  reasoning  capabilities  are  needed  to  support  such  a  co¬ 
operation.  This  is  in  the  context  of  a  new  distributed  problem  solving  (DPS)  testbed, 
DRESUN,  that  simulates  a  distributed  set  of  RESUN  interpretation  systems  and  solves 
DVMT2-like  aircraft  monitoring  problems.  DRESUN  is  composed  of  a  set  of  agents,  called 
RESUN,  that  have  interaction  capacities.  DRESUN’ s  main  goal  is  to  ensure  the  global 
consistency  of  local  agents'  solutions.  To  do  so,  one  must  address  the  distributed  coordination 
problem  (Decker  and  Lesser  1993)  which  aims  at  scheduling  tasks3  over  a  set  of  distributed 
agents  working  on  sets  of  inter-related  problems.  In  such  a  context,  uncertainty  is  still  a 
problem,  since  agents  cannot  have  complete  information  about  the  environment.  The  more 
uncertainty  we  have  about  the  environment,  the  more  uncertainty  we  get  in  coordination 
relationships.  Uncertainty  in  these  relationships  can  be  evaluated  as  the  necessary  amount  of 
communication  needed  to  reach  a  solution  to  the  problem.  For  example,  in  an  aircraft 
monitoring  scenario,  two  agents  are  responsible  for  two  intersecting  regions.  Each  agent  can 
only  receive  information  about  its  own  region.  The  goal  of  the  system  is  to  identify  the 
aircraft  that  are  moving  through  the  regions  of  interest,  to  determine  their  types  and  to  track 
them  through  the  regions.  Received  data  may  be  uncertain  because  of  noise.  In  addition, 
agent  A’s  sensors  may  fail,  while  agent  B's  sensors  are  still  operating.  Hence,  agent  B  can 
inform  agent  A  about  what  is  going  on  in  the  intersection  region. 

As  an  extension  to  the  DRESUN  testbed,  a  study  on  distributed  coordination  problem  has  led 
to  the  definition  of  five  coordination  mechanisms  needed  in  a  distributed  situation  assessment 
(DSA)  context  (Carver  and  Lesser  1995).  These  coordination  mechanisms  deal  with  updating 
non-local  viewpoints,  communicating  results,  handling  simple  redundancy,  handling  soft  and 
hard  coordination  relationships. 

In  general,  there  is  an  overlap  between  agents’  areas  of  responsibilities.  In  order  to  coordinate 
their  actions  and  plans,  agents  must  be  able  to  change  (by  themselves  or  with  the  help  of  the 
system  designer)  their  areas  of  responsibilities.  Hence,  it  is  possible  to  make  changes  in  the 
planned  tasks  and  to  exchange  roles  between  agents  if  necessary.  This  point  has  to  do  with 
the  notions  of  organisation  and  reorganisation  of  a  group  of  agents  (Decker  and  Lesser  1 995). 
To  analyse  the  performance  of  a  particular  organisation,  it  is  important  to  know  which 


2  DVMT  stands  for  Distributed  Vehicle  Monitoring  Testbed. 

3  See  (Decker  and  Lesser  1 994)  for  a  description  of  a  modelling  framework  TAEMS  used  to  represent  an 
abstract  task  environment. 
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portion  of  the  set  of  tasks  each  agent  is  likely  to  process.  Hence,  agents  organisations  may  be 
static  or  dynamic. 

Often  system  performance  can  be  significantly  improved  thanks  to  dynamic  reorganisation  of 
a  group  of  agents.  Therefore  meta-level  communication  between  agents  about  their  local 
loads  can,  with  a  small  communication  cost,  pinpoint  the  true  costs  and  the  benefits  of  the 
various  organisational  structures,  allowing  an  informed  organisational  decision  to  be  made. 
Instead  of  an  agent  making  decision  about  restructuring  or  load  balancing  by  assuming  the 
average  load  of  the  group,  the  agent  knows  the  actual  load  for  its  neighbouring  agents. 

4.3  Human  factors  in  DF  process 

The  evolution  of  computer-based  systems  that  display  intelligent  behaviour  has  had  profound 
effect  on  their  design  and  on  the  way  they  interact  with  users  (Waltz  and  Llinas  1990,  p.  45 1). 
In  designing  such  systems,  one  must  focus  on  the  following  factors: 

Problem  specification. 

Task  analysis. 

Task  allocation  (person-machine). 

Determination  of  human  informational  needs, 

Application  of  human  factors  principles  to  interface  design. 

The  key  element  to  be  considered  in  the  design  is  the  cognitive  characteristics  of  the  person- 
machine  interaction  at  the  cognitive  level.  This  depends  on  the  type  of  communication  that 
takes  place  between  the  person  and  the  machine.  Such  a  communication  may  take  several 
forms  such  as  asking  questions,  presenting  choices,  recommending  strategies,  etc. 

4.4  Conclusion 

In  this  part,  we  discussed  the  benefits  and  drawbacks  of  the  use  of  AI  techniques  in  the  Data 
Fusion  process.  The  main  techniques  provided  by  AI  technology  are  Expert  Systems,  Natural 
Language  Processing,  Plan  Recognition,  Machine  Learning,  Intelligent  Assistance  and 
Pattern  Matching.  The  use  of  one  or  more  of  these  techniques  must  deal  with  a  number  of 
requirements  such  as  real-time  constraints,  dealing  with  uncertainty  into  the  data,  time- 
varying  data  and  symbolic  and  numeric  data  combination.  In  a  distributed  context,  where  data 
is  collected  from  different  sites,  traditional  architectures  such  as  the  blackboard  architecture 
appear  as  a  bottleneck  having  a  bad  influence  on  the  whole  system  performance.  Multi-agent 
techniques  which  are  a  sub-part  of  the  Distributed  Artificial  Intelligence  (DAI)  domain  are 
well- adapted  to  deal  with  distributed  and  data-intensive  requirements. 

In  the  next  section,  we  present  a  research  programme  which  aimed  at  using  knowledge-based 
systems  to  support  the  Data  Fusion  process  in  a  Naval  application. 


5.  PART  4:  Data  Fusion:  A  Naval  Application 


5.1  Introduction 
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This  report  is  a  summary  of  a  research  programme  conducted  by  the  Exercise  Analysis  Group 
at  the  Command  and  Control  (C2)  Department  at  DRA  Postdown  (DRA  1989)  that 
investigates  the  ways  in  which  new  technologies  can  be  used  to  support  various  aspects  of 
surface  ship  command  and  control  in  the  naval  domain. 

One  of  the  major  shortfalls  in  the  ability  to  build  and  organise  good  C2  systems  is  the  lack  of 
adequate  models  of  the  whole  system  including  the  human  element.  For  this  reason,  a 
research  programme  is  in  progress  in  ARE  Postdown  that  studies  the  use  of  knowledge-based 
systems  for  solving  problems  in  the  time-critical  areas  of  naval  battle  management.  The 
purpose  of  this  programme  is  the  development  of  decision  support  systems  capable  of 
providing  tactical  advice  to  the  combat  management  team  of  a  naval  ship  or  a  group  of  ships. 
Knowledge-based  Data  Fusion  Technology  Demonstrator  System  (TDS)  has  been  designed 
in  order  to  investigate  the  functional  and  technical  aspects  associated  with  knowledge-based 
technology  and  the  operational  issues  that  ensue  from  its  use.  TDSs  interact  with  the  ship’s 
live  equipment  in  a  multi-threat  environment.  The  selected  strategy  was  to  gradually  increase 
the  complexity  of  scenarios  presented  to  the  TDSs  in  terms  of  numbers  and  variety  of 
consorts,  density  of  information  and  combination  of  threats.  TDSs  must  provide  automated 
support  to  the  air,  surface  and  sub-surface  picture  compilers  and  supervisors. 

The  prime  objectives  of  the  research  programme  are  (Byrne  et  al.  1989b): 

To  investigate  the  technical,  time-scale  and  financial  risks  associated  with  the  use  of  this 
technology; 

To  identify  the  role  of  automated  decision  support  within  naval  combat  management  and 
ascertain  the  adequacy  of  knowledge-based  technology  to  provide  such  a  support; 

To  specify  the  requirements  for  the  knowledge-based  components  of  a  combat 
management  system; 

To  explore  the  manning  issues  entailed  by  the  use  of  this  technology. 

In  the  following  sections,  we  present  the  C2  model  and  will  discuss  the  importance  of 
automating  parts  of  this  process.  We  will  also  discuss  the  problems  that  may  arise  from  such 
an  automation.  We  will  next  propose  some  partial  solutions  to  these  problems,  such  as  the  use 
of  parallel  processing  methods  and  knowledge-based  systems  and  techniques  that  can  reason 
under  uncertainty.  Finally,  we  will  emphasise  the  importance  of  human  factors  in  C2  systems 
and  will  present  a  user-machine  interface  that  meets  the  requirements  of  such  factors. 

It  should  be  noted  that  ail  the  information  collected  from  the  referenced  documents  can  also 
be  found  in  (Miles  1988). 

5.2  The  C2  model 

An  important  problem  of  command  and  control  in  the  naval  context  is  related  to  the  real-time 
decisions  that  must  be  made  in  order  to  accomplish  a  planned  mission.  A  mission  involves 
the  participation  of  a  large  number  of  ships,  aircraft  and  even  submarines  which  form  the 
Naval  Task  Group.  Sometimes,  the  term  “tactical”  is  used  to  distinguish  between  real-time 
decisions  and  higher-level  command  and  control  decisions  made  for  whole  oceans  and 
continents. 

In  (Miles  1988)  a  four-stage  model  of  command  and  control  is  presented: 
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-  Perception  -  sense  events  occurring  in  the  real-world  of  interest4. 

-  Assessment  -  assess  what  is  happening. 

Decision  —  decide  how  to  act  in  order  to  achieve  mission  goals. 

-  Direction  -  direct  resources. 

The  perception  of  occurred  events  combines  various  types  of  sensor  data  in  order  to  form  a 
more  intelligible  representation  called  the  “tactical  picture”.  This  process  of  combination  is 
called  “Data  Fusion”  (DF).  The  results  of  the  data  fusion  must  be  used  cleverly  in  order  to 
assess  what  is  happening  in  the  world  of  interest.  This  process  is  called  “Situation 
Assessment”  (SA).  Decisions  should  be  made  successively  so  that  the  mission  goals  can  be 
pursued.  These  decisions  concern  the  “Resource  Allocation”  or  re-allocation  and  aim  at 
executing  a  planned  course  of  actions.  They  also  imply  the  “Resource  planning”  if  measures 
are  to  be  taken  to  anticipate  future  situations.  Thus,  the  stages  of  naval  C2  are  data  fusion, 
situation  assessment,  resource  allocation  and  resource  planning.  In  the  following  sections,  we 
give  only  a  description  of  the  first  two  stages  that  are  of  interest  to  us. 

5.2.1  Tactical  Data  Fusion 

This  process  consists  of  the  representation  of  a  coherent  tactical  world  picture  based  on  a 
combination  of  sensor  data,  plans  and  intelligence.  The  idea  behind  the  tactical  picture  is  to 
give  a  tactically  significant  vision  of  the  objects  in  the  world  of  interest  to  a  warship:  where 
they  are,  what  they  are  and  where  they  are  going.  Concretely,  building  a  tactical  picture  is 
detecting,  locating,  tracking  and  possibly  classifying  objects  (Byrne  et  al.  1989a). 

Data  fusion  can  be  viewed  as  a  two-stage  process: 

Stage  1  -  Assemble  all  data  that  refer  to  each  individual  domain  object.  Multi-sensor  data 
arrive  through  different  channels  from  group  of  ships,  aircraft  and  possibly  submarines 
operating  collectively.  Sources  of  evidence  include  sensors  (Radar,  IFF  (Identification 
Friend  or  Foe),  ECM  (Electronic  Counter  Measures),  ESM  (Electronic  Support 
Measures),  Active  Sonar,  Passive  Sonar),  data  links  (ship  to  ship,  aircraft  to  ship,  shore 
to  ship),  intelligence  (electronic,  communications,  human),  and  other  useful  information 
such  as  plan  and  command  information  (e.g.  flight  plans  for  aircraft),  environmental  data 
(weather,  oceanographic  information,  etc.),  equipment  database  (weapon  systems,  etc.). 

Stage  2  -  Combine  assembled  data  for  each  object  to  estimate  or  infer  most  likely  values 
for  interesting  parameters.  The  main  required  platform5  parameters  in  a  model  are 
position,  velocity,  identity,  capability,  mission  and  allegiance.  Allegiance  assessment 
which  determines  whether  a  contact  has  friendly,  neutral  or  hostile  allegiance  is  a 
particular  aspect  of  the  data  fusion  problem  and  is  characterised  by  the  need  to  provide 
an  assessment  based  on  existing  information.  This  excludes  the  option  of  delaying  an 
assessment  until  further  information  is  available  or  until  conflict  of  information  is 
resolved.  Any  information  (evidence)  about  the  contact  can  be  indicative  of  its 
allegiance.  Evidence  may  take  the  following  forms  (Hirst  1989):  geographic  position  of 
the  contact,  transponder  response  (IFF,  electronic  emissions  detected  by  ESM),  visual 

4  “World  of  interest”  in  the  naval  application  consists  of  a  volume  of  space  including  the 
airspace  used  by  aircraft  and  the  ocean  space  used  by  submarines. 

5  Platform:  object  of  interest :  aircraft,  ship,  submarine... 
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identification,  threatening  and  hostile  behaviour,  and  other  factors  such  as  human 
intuition,  track  history  or  intelligence  reports. 

5. 2. 1.1  Scope  of  data  fiision 

The  results  of  tracking  and  image  recognition  are  used  as  inputs  to  the  data  fusion  system. 
The  output  of  a  data  fusion  system  may  be  directly  used  by  humans  or  may  be  input  into  a 
situation  assessment  level.  Situation  assessment  is  also  considered  by  some  people  to  be  a 
part  of  the  data  fusion  process.  Indeed,  drawing  high-level  conclusions  from  the  results  of  the 
merging  of  input  data  is  also  a  fusion  process.  Data  fusion  is  first  a  problem  of  correlation 
which  consists  in  finding  which  pieces  of  information  refer  to  the  same  real-world  object,  and 
secondly  a  problem  of  classification  (Jungert  1999;  Zhang  and  Jiao  1998),  i.e.  estimating  and 
inferring  attributes  of  the  object  from  the  assembled  evidence.  Figure  10  shows  the  scope  of 
the  data  fusion  problem  (Miles  1987b). 


Detection  DF  (wide  definition)  Resource  Allocation 


Tracking  DF  (narrow  definition)  Situation  assessment 

Figure  10.  Two  definitions  of  the  data  fusion  process. 


This  figure  shows  that  the  “wide  definition”  of  data  fusion  includes  tracking  and  situation 
assessment,  whereas  the  “narrow  definition”  includes  only  correlation  tasks. 

5.2. 1.2  Multi-platform  data  fusion 

Assuming  that  a  data  link  and  voice  communications  are  available,  it  will  be  possible  to 
collect  sensor  data  from  other  platforms  to  form  a  tactical  picture  over  a  much  wider  area 
than  the  one  perceived  directly  by  own  ship.  This  is  called  multi-platform  data  fusion. 
Because  of  the  large  volume  of  data  produced  by  active  and  passive  sensors,  data  links  are 
required  to  support  multi-platform  data  fusion.  Even  with  data  links,  the  amount  of  data 
transferred  over  a  period  of  time  (bandwidth)  will  be  limited.  The  multi-platform  data  fusion 
is  a  two-level  correlation  hierarchy.  At  the  first  level,  are  correlated  the  tracks  from  different 
platforms  (multi-tracks).  At  the  second  level,  are  correlated  the  multi-tracks  in  order  to 
identify  platforms  or  groups  of  platforms. 

5.2. 1.3  Conclusion 

To  conclude,  we  can  state  that  an  intelligent  data  fusion  system  in  a  naval  environment 
should  (Lakin  and  Miles  1989): 
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-  Effectively  cope  with  the  large  quantity  of  input  data  that  is  available  for  warship  now, 
and  the  one  that  will  be  available  in  increasing  amounts  in  the  future. 

Include  and  make  full  use  of  non-real-time  and  encyclopaedic  information  as  well  as  real¬ 
time  sensor  data. 

-  Achieve  the  reaction  times  necessary  to  rapidly  respond  to  changing  situations. 

Relieve  human  operators  of  routine  low-level  tasks. 

5.2.2  Situation  Assessment 

Situation  assessment  builds  on  the  tactical  picture  produced  by  data  fusion  in  order  to  form  a 
consistent  view  of  it.  This  is  done  by  identifying  and  prioritising  potential  threats  in  the 
environment  of  a  single  ship  or  a  group  of  ships  in  terms  of  deployments,  capabilities, 
resource  allocation,  effectiveness  of  own  and  opposing  forces.  Environmental  factors  such  as 
the  weather  and  the  geographic  and  oceanographic  conditions  affect  these  assessments.  The 
political  situation  and  the  rules  of  engagement  also  influence  the  judgements  made. 

Situation  assessment  feeds  back  conclusions  to  the  lower  levels  in  order  to  fill  in  unknowns. 
For  example,  an  aircraft  may  have  been  assessed  as  hostile,  but  others  nearby  may  have  no 
data  to  support  their  allegiance.  By  assessing  the  group  as  a  formation,  the  allegiance  can  be 
propagated  to  all  the  members  of  the  group. 

5.2.2. 1  Assessed  elements 

Those  aspects  of  the  tactical  situation  which  could  be  assessed  are: 

Threats.  They  can  be  classified  as  direct,  indirect  and  potential.  “Direct  threats”  are 
observed  hostile  units  in  the  act  of  attacking.  “Indirect  threats”  are  estimates  of  attacks 
that  observed  hostile  units  are  likely  to  make  according  to  intelligence  about  the  weapons 
they  carry.  “Potential  threats”  are  an  assessment  of  threats  which  may  be  encountered 
according  to  intelligence  only.  Potential  threats  can  be  assessed  on  a  much  longer  time 
scale  and  there  is  less  need  for  machine  assistance.  Rules  for  threat  assessment  merge  the 
currently  perceived  deployment  of  enemy  units  with  encyclopaedic  intelligence  data  on 
enemy  weapon  systems  and  tactics  to  generate  attack  scenarios.  Parameters  to  be 
estimated  include  the  type  of  attack,  its  imminence,  the  number  of  units  involved,  likely 
targets,  etc. 

-  Engagements.  They  may  determine  the  next  course  of  actions.  Unfortunately,  information 
relative  to  this  assessment  is  difficult  to  obtain  in  the  required  time  scale. 

Weapon  systems  geometry.  Evaluate  the  possible  conflicts  from  the  positioning  of  various 
weapon  systems  and  produce  alerts  if  dangerous  situations  which  may  cause  accidents  are 
discovered. 

Weapon  states. 

-  Rules  of  engagement  (ROE)  and/or  exclusion  zone  (EZ)  infringements. 

Sensor  coverage. 

Weapon  coverage. 

-  Adherence  to  plans.  Constantly  monitor  whether  the  plans  are  being  followed  and,  if  not, 
alert  the  command  about  the  discrepancies.  For  friendly  groups,  any  discrepancy  between 
the  observed  situation  and  the  planned  one  can  alert  the  user.  For  neutral  groups,  any 
uncharacteristic  behaviour  or  deviation  from  an  expected  plan  should  alert  the  command 
or  invoke  a  reassessment  of  identity  and  mission. 
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-  Each  side  perception  of  the  other.  The  outcome  of  this  assessment  can  help  determine 
whether  active  sensors  must  be  used.  This  may  give  away  more  information  to  the  enemy 
than  provide  information  to  own  forces. 

Defence  screen.  Its  aim  is  to  alert  the  command  as  soon  as  possible  about  any  weak  areas 
such  as  those  caused  by  units  changing  station,  equipment  failures  and  losses  during 
actions. 

The  main  goal  of  situation  assessment  is  to  build  a  complete  high-level  description  of  what  is 
happening  now,  given  the  present  position,  and  to  predict  what  is  likely  to  happen  in  the 
future.  Unfortunately,  only  an  incomplete  detailed-level  picture  of  the  current  situation  is 
available.  Three  complementary  strategies  are  suggested  to  bridge  the  gap  between  these  two 
levels  of  knowledge: 

Groups  formation 
Plan  recognition 
Prediction. 

5. 2. 2. 2  Groups  formation 

Group  formation  views  situation  assessment  as  an  extension  of  the  data  fusion  process.  The 
aim  of  this  process  is  to  create  a  higher-level  view  of  the  tactical  picture  from  those  produced 
by  the  data  fusion  process  for  individual  objects,  so  that  further  inferences  on  identity, 
allegiance,  function  and  mission  become  possible.  The  following  levels  of  groups,  ordered 
from  lowest  to  highest,  have  been  defined: 

Vehicles. 

-  Spatial  Groups  -  groups  of  vehicles  formed  by  spatial  cluster  analysis. 

Functional  Groups  —  groups  of  vehicles  of  the  same  hostility  carrying  out  similar 
functions  such  as  hostile  ships  performing  an  anti-surface  role.  A  vehicle  may  belong  to 
several  Functional  Groups. 

-  Interacting  Groups  -  groups  of  Functional  Groups  which  have  similar  objectives  such  as 
attacking  the  same  target.  There  are  two  important  categories  of  Interacting  Groups: 
Defensive  Groups  and  Attack  Groups. 

Own/Enemy/Neutral  Groups  —  Own  Group  is  the  group  of  all  friendly  Interacting  Groups, 
and  Enemy  Group  is  the  group  of  all  hostile  Interacting  Groups.  Neutral  Group  is  the 
group  of  all  neutral  vehicles. 

Tactical  World  State  —  ties  Own  Group  and  Enemy  Group  together. 

In  general,  the  set  of  vehicle  data  formed  by  fusing  data  from  all  sources  will  be  incomplete 
and  therefore  the  attempt  to  form  all  the  functional  and  interacting  groups  of  interest  will  not 
be  successful. 

5. 2.2.3  Plan  recognition  and  prediction 

At  a  higher  level,  some  elements  of  the  tactical  plan  being  executed  by  unknown  units  may 
be  identified.  These  elements  may  be  used  to  infer  the  missions  of  units  whose  presence  was 
previously  unexplained.  Thus  some  parts  of  the  situation  assessment  may  be  seen  as  a  “plan 
recognition”  activity.  The  idea  is  to  form  a  description  of  patterns  which  are  accepted  to  exist 
and  then  match  those  patterns  against  the  available  data.  If  a  good  match  can  be  found,  then 
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details  of  the  pattern  which  are  not  present  in  the  data  can  be  inferred  with  some  confidence 
(prediction).  This  process  is  called  plan  recognition  and  it  aims  at  predicting  unknown 
conclusions  due  to  incomplete  data.  The  plan  recognition  will  obviously  fail  when  behaviours 
that  are  outside  the  scope  of  the  stored  plans  are  observed.  If  conclusions  cannot  be  inferred 
from  incomplete  data,  the  prediction  can  also  be  made  by  simulation. 

5.3  Why  is  it  important  to  automate  in  C2? 

One  view  of  the  automation  in  C2  is  that  it  should  be  capable  of  providing  the  right 
information  at  the  right  time,  to  the  right  person,  in  the  right  form,  so  that  it  can  be 
assimilated  and  processed  in  a  time-critical  situation. 

In  designed  worlds,  such  as  in  industrial  processes  and  air  traffic  control,  the  domain 
elements  are  assumed  to  be  co-operative.  In  real  hostile  world  such  as  military  surveillance 
and  situation  appreciation,  the  elements  are  unco-operative.  Thus,  a  large  amount  of  disparate 
data  is  collected.  The  use  of  more  powerful  and  more  sophisticated  sensors  and  the  provision 
of  high  bandwidth  data  links  will  generate  an  even  greater  volume  of  data  to  be  processed. 
The  high  data  output  rates  (avalanche  of  information)  to  human  resources  result  in  an 
overload  of  information  for  decision  makers  (man-intensive  systems).  To  give  an  idea  about 
the  system’s  performance,  it  has  to  cope  with  up  to  100  messages  per  second  from  various 
data  sources.  In  addition,  in  current  C2  organisations,  there  is  a  great  deal  of  message  traffic 
in  natural  language  which  places  huge  demands  on  human  resources  just  to  assimilate  the 
facts  and  update  the  databases  (Miles  1987a). 

The  main  objectives  of  automation  in  C2  is  to: 

•  free  the  operators  from  doing  low-level  tasks  and  allow  them  to  concentrate  on  higher- 
level  tasks  such  as  decision  making  and  planning. 

•  handle  the  slow  response  rates  due  to  human  interventions.  This  slowness  results  either 
from  the  user’s  inability  to  interpret  large  amounts  of  data  in  the  required  time  or  from  his 
incapacity  to  react  in  time.  This  problem  is  related  to  the  system  interface  design. 


5.4  What  are  the  problems  engendered  by  automation  in  C2? 

The  automation  in  C2  raises  several  questions.  First,  what  are  the  models  that  should  be  used 
to  deal  with  the  real-time  requirements  and  how  is  it  possible  to  ensure  that  such  models  are 
well-suited  to  fulfil  such  requirements.  Second,  what  is  the  impact  of  such  an  automation  on 
system  users.  Should  they  trust  it?  How  can  they  interact  with  the  system?  Does  this 
interaction  affect  the  operators’  methods  of  work?  Is  the  task  appropriately  distributed 
among  the  operators  and  the  machines? 

In  the  following  sections,  we  will  argue  why  purely  algorithmic  approaches  in  automated  C2 
are  insufficient  and  inadequate.  We  will  also  discuss  the  psychological  impact  of  such  an 
automation  on  the  user. 

5.4.1  Inability  of  algorithmic  approaches  to  deal  with  symbolic  and  numeric  data 
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Data  fusion  is  a  critical  process  in  naval  C2  since  it  aims  at  providing  the  operations  room 
staff  with  accurate  tactical  pictures,  based  on  disparate  real-time  and  non-real-time  data. 
Because  of  the  large  amount  of  data  and  their  incomplete  and  fragmentary  nature,  real-time 
data  are  an  issue  for  C2.  Although  algorithmic  techniques  are  widely  used  in  data  fusion 
processes,  they  are  unable  to  cope  with  information  of  different  nature  (symbolic  and 
numeric).  In  addition,  new  communication  and  intelligence  systems  will  increase  the 
disparity  of  data  types,  particularly  the  non-real-time  categories. 

5.4.2  Inability  of  algorithmic  approaches  to  represent  high-level  knowledge 

The  naval  C2  scenarios  tend  to  become  increasingly  complex  because  of  the  sophistication  of 
sensors,  weapon  systems  and  communication  technologies.  Information  sources  include  not 
only  real-time  data  but  also  information  provided  by  human  intelligence,  encyclopaedic  and 
operational  plans,  etc.  The  inability  of  algorithmic  approaches  to  deal  with  symbolic  and 
numeric  data  makes  them  inadequate  to  encode  high-level  knowledge  necessary  in  high-level 
processes  such  as  situation  assessment,  human  messages  processing,  reasoning  and 
explanation  facilities. 


5.4.3  Inability  of  algorithmic  approaches  to  provide  high-level  support  for  operators 

Whatever  the  level  of  decision  making  may  be,  it  requires  high-level  knowledge  that  can 
provide  a  global  view  of  the  tactical  picture  or  the  current  situation.  The  encapsulated 
knowledge  in  algorithms  is  not  rich  enough  to  give  such  a  global  view,  and  even  if  it  did,  the 
available  knowledge  could  not  be  used  to  explain  how  such  a  view  was  built.  For  all  these 
reasons,  algorithmic  approaches  offer  little  support  for  decision  makers.  Yet,  this  support 
would  be  very  valuable,  especially  in  a  time-critical  context  where  actions  cannot  be  delayed 
and  decisions  must  be  made  just  in  time. 

5.4.4  Psychological  aspects 

Automated  decision  support  tools  must  be  provided  with  facilities  that  can  guarantee  user 
acceptance.  A  key  factor  is  the  design  of  the  user-computer  interface  (see  Section  6).  Another 
way  to  increase  the  user's  confidence  in  an  automated  tool  is  to  show  that  the  system  can 
solve  real  problems. 

5.5  Solutions 

In  the  following  sections,  we  present  solutions  to  the  problems  discussed  in  Sections  4  and  5. 
These  solutions  are  basically  related  to  the  real-time  operational  performances  of  parallel 
computing  techniques,  the  high-level  knowledge  representation  capabilities  of  knowledge- 
based  systems  and  finally,  the  technical  means  to  handle  uncertainty. 

5.5.1  Parallel  processing  techniques 
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To  meet  the  real-time  requirements,  parallel-processing-based  approaches  have  been 
investigated.  These  approaches  attempt  to  provide  short-term  and  long-term  solutions  for  C2 
systems. 

Connectionist  architectures  have  also  been  studied: 

A  first  type  of  connectionist  architecture  uses  local  representations.  Each  element 
represents  a  specific  concept  and  each  link  represents  a  specific  type  of  relation.  The 
elements  are  connected  into  a  semantic  network. 

Another  type  of  architecture  uses  neural  networks  and  tries  to  recognise  patterns  through 
their  dynamic  behaviour.  These  methods  are  still  limited  because  of  their  slowness  in 
simulation. 

A  more  likely  future  solution  may  be  a  form  of  what  are  generally  called  “Massively  Parallel 
Architectures”,  one  of  which  is  the  NETL  machine  (Fahlman  1979).  The  NETL  machine  uses 
a  semantic  network  to  represent  real-world  knowledge.  This  representation  is  very  general.  In 
principle,  it  would  be  possible  to  represent  the  entire  knowledge  about  the  military  world, 
both  encyclopaedic  and  real-time.  However,  this  would  require  a  very  large  number  of  nodes 
and  links.  Searching  through  the  network  could  become  very  time  consuming  on  a 
conventional  computer.  Yet,  the  NETL  machine  uses  a  simple  processor  for  each  node  and 
each  link  of  the  semantic  network,  making  it  possible  to  do  the  search  with  total  parallelism. 

5.5.2  Use  of  knowledge-based  systems  and  expert  systems 

Conventional  computing  techniques  have  shown  little  success  in  formulating,  maintaining 
and  interpreting  a  tactical  picture  resulting  from  data  fusion.  Hence,  there  is  an  increasing 
interest  in  the  possible  use  of  knowledge-based  systems  in  general  and  expert  systems,  in 
particular.  These  systems  can  be  used  to  deal  with  unstructured  problems  and  they  have  some 
abilities  to  explain  their  reasoning. 

However,  the  real-time  processing  requirements  must  be  kept  in  mind  and  one  must  be  sure 
that  the  knowledge-based  model  can  be  combined  with  parallel  processing  techniques 
(see  Section  5.1).  In  this  study  it  was  showed  that  in  a  rule-based  system  there  are  set  of 
production  rules  which  are  independent.  The  static  analysis  identified  the  set  of  rules  which 
access  the  same  type  of  data.  The  dynamic  analysis  showed  that  rules  belonging  to  different 
sets  and  firable  at  the  same  time  can  be  processed  in  parallel. 

At  the  data  fusion  level  (see  Section  2.1),  rules  may  be  used  in  a  forward-chaining  reasoning 
to  handle  the  multi-sensor  data  correlation  problem.  With  regard  to  the  situation  assessment 
(see  Section  2.2),  rules  are  required  for. 

-  Group  Formation: 

-  Forming  and  maintaining  groups; 

-  Correlating  group  evidence; 

-  Inferring  group  parameters  and  propagating  to  individuals. 

Plan  Recognition  and  Prediction:  this  would  normally  be  organised  as  a  backward¬ 
chaining  task  in  a  rule-based  structure,  in  contrast  to  the  forward-chaining  nature  of  the 
data  fusion  activity. 
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In  addition,  knowledge-based  systems  can  gain  the  user’s  confidence  by  providing 
explanation  facilities  (see  Section  5. 5.2.4). 

5. 5. 2. 1  Architectures 

In  the  research  programme  undertaken  by  the  ARE  (Admiralty  Research  Establishment  of 
Postdown),  a  blackboard-based  expert  system  was  used  to  perform  data  fusion.  The  main  data 
structure  stored  in  the  blackboard  is  a  hierarchy  of  hypotheses.  Each  hypothesis  represents  a 
possible  conclusion  for  the  group  of  hypotheses  situated  at  a  lower  level.  Input  data  are 
posted  as  new  hypotheses  on  the  blackboard.  Experts  are  represented  by  a  set  of  rules,  called 
knowledge  sources,  that  manage  the  hierarchy  of  hypotheses.  A  scheduling  mechanism  is 
used  in  order  to  ensure  that  the  most  appropriate  knowledge  is  applied  (opportunistic 
reasoning).  SPL  International  was  sponsored  to  design  and  build  a  general  purpose 
blackboard  framework  which  is  now  called  MXA  (Multiple  eXpert  Architecture).  MXA 
includes  various  features: 

-  A  language  for  expressing  rules; 

-  A  hypothesis  structure; 

-  An  inference  engine,  i.e.  the  knowledge  source  control  program; 

-  Explanation  generation  capabilities. 

Scheduling  is  the  process  that,  given  the  current  state  of  the  blackboard  and  any  available 
input  data,  determines  which  knowledge  source  is  to  be  invoked  next.  In  MXA,  the 
scheduling  of  knowledge  sources  uses  a  rule-based  approach.  The  rules  for  scheduling  are 
held  in  Meta-Knowledge  Sources  and  hypotheses  are  described  by  declarations. 

In  order  to  evaluate  different  systems’  performances,  the  blackboard  architecture  was 
compared  to  other  architectures  (Miles  1989): 

A  production  system  using  ART  (on  Symbolics  3600  LISP  machine). 

A  hybrid  production  system  /  imperative  programming  approach  using  MUSE  (on  Sun  4 
Workstation). 

-  The  MXA  blackboard  architecture  using  Ada  (on  Microvax  3500  workstation)  rather  than 
an  AI  toolkit  because  of  the  real-time  performance  required. 

Let  us  point  out  that  the  different  machines  used  to  test  these  architectures  have  an  equal 
performance. 

5.5. 2. 2  Levels  of  knowledge 

The  heart  of  the  data  fusion  system  is  a  knowledge-based  system  which  formulates  a  coherent 
tactical  picture  by  fusing  together  in  real  time  all  the  available  information  on  board  a  ship 
that  has  been  conveyed  from  disparate  sources.  Using  a  knowledge-based  approach, 
demonstrators  have  been  developed  which  are  capable  of  generating  an  on-going  tactical 
picture  not  only  from  a  wide  variety  of  simulated  data  types,  but  also  from  data  recorded  at 
sea  during  naval  exercises.  While  the  first  generation  of  demonstrators  addresses  only  the 
first-level  task  of  picture  compilation  through  multi-platform  multi-source  data  fusion,  the 
second  generation  includes  the  second-level  task  of  tactical  situation  assessment  and  the 
third-level  task  of  formulating  an  appropriate  response.  Because  knowledge  involved  in  the 
situation  assessment  process  is  more  complex,  the  knowledge  acquisition  and  encapsulation 
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of  naval  expertise  into  rules  are  not  straightforward.  Interviews  have  been  used  to  elicit  this 
knowledge. 

5.5. 2. 3  Knowledge  types 

The  knowledge-based  system  that  was  used  in  the  data  fusion  and  the  situation  assessment 
processes,  identified  five  categories  of  knowledge  sources: 

The  tactical  picture; 

Geographical  and  meteorological  information; 

Tactical  knowledge; 

Encyclopaedic  knowledge; 

The  expertise  of  the  expert. 

The  database  must  not  only  represent  information  relative  to  tracked  platforms,  but  also 
higher- level  information,  such  as  patterns  of  behaviour,  necessary  for  the  situation 
assessment  process.  In  order  to  create  a  more  intelligent  C2  system,  a  multi-level  data 
representation  is  required  (see  Figure  1 1  for  a  simplified  example). 


To  support  the  situation  assessment  process,  a  more  complex  multi-level  blackboard-data 
representation  has  been  designed.  A  bottom-up  presentation  of  the  hypotheses  hierarchy  is 
given  below  (Lakin  et  al.  1989): 

Tracks,  Plans  &  Reports 
Possible  correlations 
Multi-sensor  tracks 
Possible  correlations 
Vehicles 

Spatial  vehicle  groups 
Functional  vehicle  groups 
Interacting  Vehicle  groups 
Hostile/Friendly/Neutral  groups 
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Rules 
IF  fast  track 

THEN  aircraft  or  missile 

IF  radar  &  ESM  on  same  bearing 
THEN  possibly  same  target 

IF  radar  type  A 
THEN  platform  X  or  Y 

IF  scanning 

THEN  surveillance  or  acquisition 


Hypothesis  Structure 


Figure  1 1.  A  simplified  example  of  the  hypothesis  structure  and  the 
rules  that  support  this  structure. 


Hypotheses  are  about: 

-  Correlations  between  tracks  from  different  sensors 

Platforms  with  estimated  parameters  from  accumulated  evidence 
Groups  of  functionality  related  to  platforms 

In  order  to  predict  the  future  from  the  current  situation,  a  history  or  script  is  required  for  each 
platform.  Moreover,  in  order  to  discard  out-of-date  data  and  make  knowledge  more  flexible,  a 
time  limit  is  set  for  these  data,  beyond  which  they  are  no  longer  considered  for  correlation. 
This  would  allow  for  data  to  be  retained  as  long  as  required  depending  on  their  importance. 


5. 5. 2. 4  Explanation  ability 

Explanation  is  defined  as  something  that  makes  one’s  view  more  coherent  and  intelligible. 
Reasoning  can  be  regarded  as  maximising  the  explanatory  connectedness  or  coherence  of 
one’s  belief. 

In  (Hirst  1989)  explanatory  structures,  called  e-structures,  were  defined  to  show  how  the 
presence  of  a  fact  can  be  explained.  For  example,  the  fact  that  an  IFF  transponder  response  is 
friendly  can  be  explained  by  the  fact  that  the  contact  is  friendly,  because  there  is  a  rule  that 
specifies  that  if  the  contact  is  friendly  then  the  response  of  IFF  is  friendly.  There  may  be 
several  competing  e-structures  for  a  given  fact. 

In  general,  the  approach  requires  principles  for: 

Determining  the  possible  e-structures; 

Choosing  the  best  explanation  from  competing  e-structures. 

The  criteria  for  choosing  the  best  e-structure  and  thus  providing  the  best  explanation  are  not 
yet  well  defined.  However,  some  general  principles  such  as  “preferring  e-structures  that 
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provide  the  most  possible  explanations”,  simplicity,  conservation  and  plausibility  have  been 
stipulated. 

A  mere  comparison  of  e-structures  will  give  rise  to  ambiguity  because  of  the  possibility  of  a 
combination  of  conflicting  propositions.  Hence,  there  is  a  need  for  further  levels  of 
explanation.  An  alternative  approach  would  be  to  assign  a  penalty  value  to  each  explanation, 
based  on  the  plausibility  of  the  additional  propositions  involved.  Thus,  unlikely  or  non¬ 
preferred  explanations  will  be  given  a  high  number  and  preferred  explanations  a  low  number. 

Explanation-based  reasoning  allows  one  to  draw  inferences  from  incomplete  information  by 
using  default  rules,  and  from  conflicting  information  by  invoking  competing  explanatory- 
structures  with  different  explanatory  power. 

5. 5. 2. 5  Problems 

Despite  all  the  advantages  of  using  a  knowledge-based  approach  in  the  data  fusion  and 
situation  assessment  processes,  many  problems  still  remain.  In  the  following,  we  present 
some  of  the  questions  that  K-B  approaches  should  address  (Anderson  et  al.  1989): 

How  can  we  deal  with  the  complexity  of  thousands  of  separate  functions  required  in  naval 
command  and  control  system? 

How  can  we  deal  with  inaccurate  and  uncertain  information  (explosion  of  possibilities)? 

-  Knowledge-based  systems  are  slow.  What  can  we  do  in  a  situation  where  real-time 
tracking  functions  are  required? 

Knowledge-based  systems  should  be  able  to  provide  concise  information  in  an  acceptable 
time  so  that  actions  are  not  delayed.  Which  tools  are  needed  to  provide  such  capability? 
Creating  knowledge-bases  demands  experts'  involvement.  It  is  difficult  to  get  the  naval 
officers  involved.  How  can  we  solve  this  problem? 

How  can  we  manage  the  maintenance  of  the  rule  base  given  that  the  system  has  to  deal 
with  new  threats  and  scenarios  and  unforeseen  situations  that  will  modify  the  stored 
knowledge?  In  the  military  domain,  it  is  very  difficult  to  update  the  knowledge  base  in  a 
laboratory  or  a  factory  because  knowledge  emerges  when  the  system  is  fielded?  (Miles 
1987a). 

-  If  the  system  is  intended  to  reason  about  battle  situations,  how  can  we  make  sure  that  it 
will  be  effective  or  even  useful  in  a  future  conflict,  given  the  difficulty  to  predict  what  is 
likely  to  happen? 

Given  that  knowledge  is  acquired  over  a  long  period  of  time,  how  can  we  deal  with  the 
time-critical  aspects  of  the  command  and  control  system? 

How  can  we  upgrade  the  knowledge-base,  while  keeping  the  system  running,  as  required 
during  critical  periods  such  as  war? 

-  If  all  ships  were  equipped  with  knowledge-based  systems,  there  is  a  danger  that  the 
conclusions  of  one  system,  based  on  tenuous  data,  be  taken  as  firm  conclusions  by 
another,  unless  the  reasoning  is  also  transferred.  Can  current  data-links  manage  such  a 
level  of  communication? 

-  The  knowledge-base  will  contain  the  warfare  policy,  the  rules  of  engagement  and  the 
tactical  doctrine  of  the  Royal  Navy.  In  principle,  any  user  could  view,  through  the 
medium  of  the  explanation  facility,  the  entire  knowledge  base.  What  level  of  explanation 
is  adequate  for  which  user? 
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If  we  incorporate  knowledge  into  a  computer  system  and  rely  on  its  expertise,  then  how 
can  we  avoid  the  danger  of  having  it  defeated  by  the  enemy’s  superior  human 
intelligence? 

-  Knowing  that  the  main  goal  of  using  a  knowledge-based  system  is  to  let  men  concentrate 
on  high-level  tasks  that  require  human  intervention,  such  as  decision  making,  then  who 
would  be  the  operational  users  of  these  automated  decision  support  tools? 

If  the  lower  levels  of  the  team  are  to  disappear,  how  do  we  train  people  to  assume  higher 
responsibilities? 

Is  it  possible  for  the  user  to  assist  the  machine  with  his  more  up-to-the-minute  knowledge 
in  real-time  context? 

It  is  difficult  to  give  precise  answers  to  all  of  these  questions.  The  solutions  depend  on 
various  contextual  and  environmental  constraints  and  as  well  as  their  correlations. 

5.5.3  Reasoning  under  uncertainty 

Parallel  processing  techniques  meet  the  real-time  processing  requirements  and  knowledge- 
based  systems  are  able  to  manage  high-level  knowledge.  Yet,  the  problem  of  dealing  with 
uncertain  information  remains  (Cohen  1985;  Li  et  al.  1999;  Rogova  and  Losiewicz  1999). 
The  data  is  generally  incomplete,  inaccurate,  ambiguous,  conflicting  and  subject  to  deliberate 
interference  and  deception  by  the  enemy.  For  example,  sometimes  we  may  have  poor 
positional  information  (uncertainty  of  positional  data  increases  rapidly  with  time  for  fast 
moving  objects),  but  a  good  identity  information.  This  leads  to  considerable  ambiguity.  In  the 
absence  of  certainty,  it  is  important  to  combine  and  correlate  data  from  different  sensors  in 
order  to  strengthen  available  pieces  of  evidence.  It  is  also  important  to  correlate  information 
received  by  the  same  sensor  at  different  times  in  order  to  maintain  interference-free  measures. 
Many  of  the  early  expert  systems  attempted  reasoning  with  uncertainty.  The  methods  used  to 
handle  uncertainty  are  of  two  types:  numerical  methods  (probability  theory,  Shafer-Dempster 
theory  of  evidence)  and  non-numerical  methods  (endorsements,  default  reasoning).  We 
review  these  methods  in  the  following  sub-sections. 

5. 5. 3.1  Numerical  Uncertainty  Schemes 

The  objective  of  numerical  uncertainty  schemes  is  to  use  a  standard  method  to  represent 
uncertainty  (e.g.  probabilistic  methods)  and  a  standard  method  to  combine  uncertainty  values 
when  conclusions  are  drawn  from  sub-conclusions  or  a  combination  of  evidence  .  Numerical 
schemes  are  difficult  to  apply  in  real-world  problems  and,  in  some  cases,  may  not  contribute 
in  a  significant  way  to  the  analysis  of  results,  compared  to  a  simple  logical  approach.  There 
are  perhaps  two  contexts  where  a  numerical  approach  can  be  used: 

When  the  statistics  of  the  input  evidence  are  well-known; 

When  a  large  number  of  test  examples  are  available. 

Another  problem  with  numerical  uncertainty  schemes  is  the  difficulty  of  explaining  the 
reasoning  to  humans.  Uncertainty  is  usually  expressed  by  humans  in  words  rather  than  in 
numerical  quantities. 

5. 5. 3. 2  Symbolic  Uncertainty  Schemes 
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Cohen  and  Grinberg  have  invented  a  method  for  representing  symbolic  uncertainty,  called 
the  “theory  of  endorsements”  (Cohen  1984;  Cohen  1985;  Cohen  and  Grinberg  1983).  These 
authors  suggest  a  method  of  inference  that  retains  a  set  of  endorsements  for  all  the  factors  that 
contribute  to  belief  or  disbelief  in  a  hypothesis.  Endorsements  are  rules  that  can  be  directly 
translated  into  statements  of  uncertainty  about  the  hypothesis  (which  appears  in  the 
conclusion  of  the  rule),  thus  providing  built-in  explanations. 

5.6  Man-machine  interface  issues 

In  the  previous  sections,  we  discussed  the  importance  of  automating  various  functions  of  the 
C2  processes  and  the  potential  problems  that  may  result  from  such  an  automation.  C2  is  a 
difficult  task  because  it  deals  with  a  large  amount  of  real-time  information  in  an 
unpredictable  environment.  For  this  reason,  human  intervention  is  always  required  and 
computer  solutions  are  insufficient.  Whatever  the  solutions  may  be,  it  is  necessary  to  have 
appropriate  user  interfaces.  In  the  next  sub-sections  we  review  several  factors  that  must  be 
taken  into  consideration  for  the  interface  design. 

5.6.1  Human  factor 

The  human  factor  is  the  most  important  issue.  It  pertains  to  the  kind  of  relationship  that  must 
be  established  between  the  machine  and  the  user: 

-  Who  is  allowed  to  use  the  system,  given  that  part  of  its  information  is  of  a  confidential 
nature.  To  which  extent  can  the  user  query  the  system? 

Given  the  real-time  nature  of  the  missions,  what  kind  of  user-machine  interactions  are 
possible?  What  is  the  role  of  the  user  and  the  machine  regarding  each  other? 

The  Command  and  Control  process  involves  both  men  and  machines  and  an  important 
amount  of  communication  between  them.  This  leads  to  the  definition  of  C3I  (Command, 
Control  and  Communication  Intelligence). 

What  should  the  system  present  to  the  user?  In  many  situations,  the  action  plan  should 
not  be  predictable  by  the  enemy.  Therefore,  the  system  should  present  to  the  user  a  set  of 
solutions  that  will  hide  the  plan  details  until  the  last  moment  when  the  user  chooses  a 
specific  solution.  Each  time  the  user  selects  an  option,  he  must  be  informed  of  its 
consequences  and  must  be  able  to  explore  those  options  that  the  system  apparently  did  not 
consider  (WHAT-IF  capability). 

5.6.2  User-machine  interactions 

The  following  list  includes  several  kinds  of  possible  user-machine  interactions: 

User  input  data.  All  kind  of  data  that  the  user  may  communicate  to  the  system  by 
different  ways. 

User  requests  for  information.  All  kind  of  requests  for  data,  explanations  and  alerts. 

User  decisions/overrides.  The  user  must  be  able  to  communicate  his  decisions  to  the 
system  and  override  the  system’s  decisions  when  necessary. 

-  Machine  output  data.  The  machine  must  be  able  to  communicate  all  its  decisions  or 
alarms  to  the  user. 

Machine  requests  for  information.  In  order  to  make  a  decision,  the  machine  may  need 
additional  data  or  explanations. 
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-  Machine  requests  for  decisions.  When  the  machine  is  unable  to  make  a  decision,  it  may 
leave  it  to  the  user. 

User  changes  to  databases. 

User  changes  to  parameters. 

User  changes  to  plans. 

User  changes  to  rules. 

The  user-machine  interactions  lead  to  the  definition  of  three  categories  of  systems: 

-  Autonomous  -  basically  a  black  box  with  minimal  interaction. 

-  Supportive  -  provides  an  interactive  framework  for  the  task  with  some  built-in 
knowledge. 

-  Collaborative  -  performs  the  task  and  provides  for  flexible  human  interaction  including 
control. 

5.6.3  Role-based  models 

The  different  kinds  of  possible  interactions  between  the  user  and  the  machine  imply  role 
definitions.  (Miles  1988,  p.  215)  defines  five  roles: 

Monitoring  role.  The  user  may  control  the  inputs  and  outputs  of  the  system.  To  do  so,  he 
or  she  needs  the  following  elements: 

Top-level  output  conclusions; 

Intermediate  levels  of  conclusion; 

The  input  evidence; 

Explanation  of  the  reasoning  leading  to  any  conclusion  at  any  level; 

Explanation  of  the  reasoning  behind  the  current  strategy  of  inference; 

A  commentary  on  significant  new  conclusions; 

A  commentary  on  the  current  inference. 

Correction  role.  This  role  is  an  extension  of  the  monitoring  role.  During  the  monitoring 
task,  the  user  may  want  to  add  his  own  ideas  to  the  domain  model.  It  is  therefore 
important  that  the  system  accepts  manual  input  while  maintaining  the  logical  consistency 
of  the  model.  The  system  must  check  whether  conflicts  arise  once  it  has  updated  the 
knowledge-base.  The  explanation  enables  the  user  to  recall  the  reason  for  which  he  made 
the  change. 

Take-over  a  low-level  task.  In  this  role,  the  user  has  the  responsibility  for  performing  the 
task  and  the  system  adopts  a  background  monitoring  role.  This  is  the  opposite  of  the 
monitoring  role. 

High-level  decision  making.  Users  may  want  to  be  involved  in  high-level  decisions  such 
as  changing  the  deployment  of  resources.  The  system  should  allow  for  this  by  adopting 
an  advisory  role,  for  example  by  setting  out  the  possible  options.  It  should  also  be  able  to 
predict  the  outcome  for  any  selected  course  of  action. 

-  Control  of  resources.  Explicit  control  may  be  provided  by  allowing  the  users  to  adjust 
scheduling  priorities,  particularly  when  overload  conditions  arise. 

5.7  Man-Machine  Interface  in  the  DF  and  SA  laboratory  demonstrator 
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Whatever  the  complexity  of  a  C2  system  may  be,  its  enhanced  capabilities  can  only  be  fully 
exploited  if  a  clear,  user-friendly  interface  is  provided.  The  function  of  the  TDS  (Technology 
Demonstrator  Systems)  is  to  capture  data  from  various  sources,  build  them  into  a  tactical 
picture  and  present  the  picture  to  the  operator  in  a  suitable  form  using  texts  and  graphics.  The 
symbols  used  to  represent  platforms  change  according  to  the  type  of  the  assessed  platform. 
Colours  are  also  used  to  show  platforms’  allegiance:  Red  (hostile),  Blue  (friendly),  Green 
(neutral)  and  White  (unknown). 

The  operator  can  click  on  any  object  for  information  (identity,  speed,  etc.).  If  further 
information  is  needed,  knowledge-based  system  capabilities  are  exploited  to  explain  why  the 
system  believes  a  specific  fact.  For  example,  the  system  may  estimate  that  an  aircraft  is 
hostile  although  the  operator  had  no  information  about  its  allegiance.  The  system  must  be 
able  to  explain  its  reasoning:  given  its  behavioural  patterns,  the  aircraft  may  be  assigned  to  a 
group  of  platforms,  in  which  case  it  takes  the  same  allegiance  as  the  group. 

In  the  next  sections,  we  present  the  three  phases  of  the  ARE  (Admiralty  Research 
Establishment  of  Postdown)  programme  on  user-machine  issues. 

5.7.1  Phase  one 

The  functionality  of  phase  one  demonstrator  (Montgomery  and  Byrne  1989)  was  limited  to  a 
single  platform,  multi-sensor  data  fusion.  This  system  had  a  simple  interface  with  four 
colours:  Red  (hostile),  Blue  (friendly),  White  (unknown),  Green  (neutral).  The  symbols  were 
oriented  as  to  depict  the  course  of  the  platforms.  Velocity  was  represented  by  an  arrow  whose 
length  was  proportional  to  the  speed  of  the  platform. 

This  system  had  several  limitations.  The  simulated  scenarios  were  slower  than  desired 
because  of  the  embedded  control  codes  which  were  sent  to  the  graphics  hardware  by  the 
display  rules.  In  addition,  the  limited  use  of  colours  (only  four  colours  were  available)  was  a 
source  of  ambiguity. 

5. 7.2  Phase  two 

The  data  fusion  demonstrator  included  a  wider  spectrum  of  input  data  and  support  data-links 
(Montgomery  and  Byme  1989).  The  multiple-window  environment  was  also  adopted  in  order 
to  increase  flexibility  in  the  presentation  of  simultaneous  information.  A  high  resolution, 
large  screen,  video  projector  was  added. 

5.7.3  Phase  three 

This  phase  (Montgomery  and  Byme  1989)  dealt  with  situation  assessment.  The  blackboard 
hypothesis  hierarchy  was  expanded  to  cover  both  data  fusion  and  situation  assessment,  the 
strategy  for  situation  assessment  being  the  group  formation.  A  graphical  solution  was  adopted 
to  show  functional  and  interacting  groups.  Also,  a  text  window  was  used  to  show  the 
additional  details  of  the  assessment:  a  threat  list  and  a  check  list  of  defensive  and  offensive 
courses  of  action. 

This  user-machine  interface  for  situation  assessment  was  not  straightforward.  It  proved  to  be 
much  more  thought-provoking  than  expected  and  required  much  more  feedback  from  the 
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users  group.  The  system’s  difficulty  in  presenting  the  results  of  the  situation  assessment  was 
reflected  in  the  comments  received  about  the  user-machine  interface.  Users  criticised  several 
aspects  of  the  interface:  the  kind  of  symbols  used  to  represent  groups  of  platforms,  the  items 
of  the  threat  list,  and  the  options  they  were  given  to  respond  to  the  threats.  In  fact,  the 
interface  was  not  customised  to  support  any  specific  role.  Moreover,  the  warfare  officers 
should  have  the  option  of  altering  the  threat  list  and  this  was  not  the  case. 

5.8  Simulations 

Scenarios  based  on  simulated  data  were  used  to  test  the  environment.  The  purpose  was  not  to 
implement  a  full  scale  system,  but  to  have  a  situation  assessment  system  for  two 
representative  operational  environments  under  warfare  conditions.  The  data  were  gradually 
updated  as  real  data,  recorded  during  missions,  were  introduced.  One  of  the  scenarios  was  a 
one-hour  sequence  from  a  four-day  NATO  maritime  exercise.  This  example  showed  how  to 
correlate  the  detections  of  different  ships  by  triangulation.  The  system’s  interface  displayed 
information  relative  to  an  object  selected  on  the  screen  and  used  different  symbols  and 
colours. 

The  ultimate  objective  of  data  fusion  is  to  combine  data  from  several  sources.  This  requires  a 
very  large  knowledge-base.  Also,  the  demonstrators’  representation  mode  implies  the  use  of  a 
very  large  number  of  rules.  This  is  why  only  a  subset  of  the  knowledge-base  was  included  in 
the  demonstration  prototype. 

5.9  Conclusion 

To  conclude,  we  can  say  that  architectures  for  real-time  knowledge-based  C2  applications 
must  meet  two  important  criteria:  a  natural  way  of  acquiring  and  representing  knowledge,  and 
real-time  performance. 

Because  in  this  model  the  levels  of  Data  Fusion  are  not  sharply  separated,  a  knowledge-based 
approach  can  be  used  not  only  for  high-level  processes,  such  as  situation  and  threat 
assessment,  but  also  for  lower  levels,  such  as  correlating  the  results  of  the  data  fusion  process 
(Cowden  1995). 

The  facts  relative  to  both  the  low  and  high  levels  of  the  C2  process  in  a  knowledge-based 
approach  show  the  importance  of  having  a  global  view  of  the  situation  in  order  to  build  what 
is  called  a  “Situation  Awareness”. 


6.  Discussion 

In  conclusion,  let  us  remark  that  in  the  specialised  literature,  the  DF  process  is  usually 
decomposed  into  levels  which  are  similar  to  those  presented  in  the  three  models  that  we 
briefly  described  above.  Researchers  agree  to  distinguish  3  levels:  a  first  level,  where  data  are 
acquired  from  multiple  sources,  filtered  and  processed  in  order  to  characterise  the  situation  to 
be  analysed;  a  second  level,  often  called  “situation  /  abstraction  assessment”,  which  deals 
with  the  elements  and  relationships  that  characterise  the  situation  and  that  are  relevant  for  the 
decision  process;  a  third  level,  often  called  “threat  assessment”,  which  aims  at  qualifying  the 
situation  in  terms  of  possible  threats  and  the  way  to  counter  them. 
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Level  1  (data  acquisition,  filtering  and  consolidation)  has  been  widely  studied  and  supported 
by  several  tools.  Level  2,  the  situation  assessment,  is  more  difficult  to  tackle  since  there  are 
many  ways  of  interpreting  data  in  a  given  context.  The  incompleteness  of  the  information 
obtained  from  the  acquisition/consolidation  level  results  into  uncertainties  that  make  the 
situation  assessment  process  a  complex  activity  to  automate.  Getting  an  acceptable  analysis 
of  the  situation  in  “quasi-real-time”  is  also  an  important  challenge,  especially  when  the 
number  of  elements  to  be  considered  increases.  Consequently,  threat  assessment  (Level  3) 
also  becomes  difficult  to  model  and  automate. 

It  is  difficult  to  provide  generic  solutions  to  the  SA  process,  because  understanding  a 
situation  seems  to  be  highly  domain-dependent.  Think  for  example  of  how  the  various 
elements  (and  their  relations)  that  characterise  a  situation  can  be  identified,  or  how  the 
elements  that  will  enhance  the  person's  understanding  of  a  situation  can  be  distinguished.  In 
practice,  however,  people  (in  our  case  the  operation  Room  Officer)  are  skilled  enough  to 
"apprehend"  complex  situations  and  to  make  decisions  in  consequence.  This  suggests  that 
getting  a  better  understanding  of  how  specialists  apprehend  and  analyse  situations  could 
provide  a  means  to  improve  the  characteristics  of  the  systems  developed  to  support  the  DF 
process.  Interestingly  enough,  we  have  found  in  the  open  literature  a  few  works  that  deal  with 
the  cognitive  aspects  of  the  SA  processes. 
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1.  Introduction  on  Situation  Awareness 

In  this  Section,  we  present  an  overview  of  several  recent  works  done  on  Situation  Awareness. 
Let  us  remark  that  we  found  in  the  literature  two  terms:  situation  awareness  and  situational 
awareness.  "Situational  awareness"  gives  better  results  when  doing  web  searches. 

Research  on  awareness  in  the  military  domain  originated  in  the  study  of  military  aviation 
where  pilots  interact  with  highly  dynamic,  information-rich  environments.  More  recently, 
researchers  have  expanded  their  interests  to  other  environments  where  situation  awareness 
plays  a  major  role,  such  as  commercial  aviation,  air  traffic  control  and  anesthesiology.  All 
these  contexts  share  the  characteristics  of  dynamism,  complexity,  high  information  load, 
variable  workload  and  risk  (Gutwin  and  Greenberg  1999). 

Awareness  is  knowledge  about  the  state  of  some  environment,  a  setting  bounded  in  time  and 
space.  (Gutwin  and  Greenberg  1999). 

There  are  several  definitions  of  situation  awareness  in  the  literature. 

Situation  awareness  is  "the  up-to-the  minute  cognizance  required  to  operate  and  maintain  a 
system"  (Adams  et  al.  1995). 

Here  is  the  definition  of  situation  awareness  given  by  Endsley  (Endsley  1987;  Endsley  1988) 
and  which  is  widely  used  in  the  literature. 

Situation  awareness  is  the  perception  of  the  elements  in  the  environment  within  a 
volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of 
their  status  in  the  near  future. 

Another  definition  of  situation  awareness  is  included  in  several  US  Navy  specifications: 
"Operator  Situation  awareness  is  comprised  of  detecting  information  in  the  environment, 
processing  the  information  with  relevant  knowledge  to  create  a  mental  picture  of  the  current 
situation,  and  acting  on  this  picture  to  make  a  decision  or  explore  further". 

Showing  the  importance  of  such  issues,  the  Situational  Awareness  Integrated  Product  Team 
(SA  IPT)  has  been  formed  in  the  US  in  response  to  fleet  tactical  aircraft  aviators  ranking 
situation  awareness  as  a  critical  mission  concern  (Gamer  and  Assenmacher  1997).  SA  IPT 
held  several  symposia  in  order  to  provide  a  forum  for  information  exchanges  about  situation 
awareness  between  academia,  military  researchers  and  industry  (Annual  Symposia  on 
Situational  Awareness,  1996-1999). 

On  the  web  we  can  find  the  SABRE  Bulletin  Board  where  people  interested  in  issues  related  to 
situational  awareness  can  exchange  ideas 

(http://users.ox.ac.uk/~pemb0595/wwwboard/faq.html) 

It  is  beyond  the  scope  of  the  present  document  to  review  all  the  works  dealing  with  situation 
awareness.  We  summarize  here  certain  key  areas. 

Situation  awareness  is  recognized  as  a  major  issue  by  the  human  factors  community  which  is 
primarily  concerned  with  techniques  for  training  people  (see  for  example  (Bass  et  al.  1997)) 
and  the  ergonomics  of  person/machine  interfaces.  Numerous  papers  are  found  on  these 
subjects  (see  bibliography).  For  example,  the  Third  Symposium  on  Situational  Awareness 
(1998)  was  devoted  to  performance  measurement  techniques  and  various  issues  of  situation 
awareness  in  relation  to  interfaces  (spatial  awareness  interfaces,  cognitive  interfaces,  intuitive 
interfaces,  multi-modal  interfaces).  So-called  "moderators  of  situation  awareness"  include 
cognitive  abilities,  motivational  states  and  technological  advances.  Several  authors  have 
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emphasized  the  fact  that  introducing  new  technology  of  greater  complexity  in  the  operators' 
workplace  may  result  in  decrease  of  operators'  situation  awareness. 

Numerous  works  deal  with  the  creation  or  enhancement  of  various  kinds  of  devices  (displays, 
sensors,  googles,  etc.)  in  order  to  enhance  operators'  awareness  in  various  domains.  For 
example,  you  can  look  at  the  proceedings  of  the  Second  Symposium  on  Situational 
Awareness  (1997)  and  at  a  large  proportion  of  references  obtained  after  a  web  search  on 
"situational  awareness".  Certain  training  systems  are  especially  built  in  order  to  help  operators 
to  enhance  their  skills  relative  to  situation  awareness;  debriefing  systems  (Bass  et  al.  1998), 
intelligent  tutoring  systems  (Bass  1998).  Other  systems  are  built  in  order  to  cooperate  with 
operators  in  order  to  help  them  to  create  and  maintain  situation  awareness  such  as  the  cockpit 
assistant  system  CASSY  (Onken  1997). 

Most  works  have  been  targeted  to  situation  awareness  of  individual  persons.  However,  in 
most  organizations  several  activities  are  performed  by  groups  of  people.  Hence,  team 
situational  awareness  becomes  an  important  issue.  Until  recently,  few  works  have  been  done 
on  this  subject.  Some  recent  works  address  the  problem  of  understanding  team  situational 
awareness  in  order  to  train  team  members  (Stout  and  Salas  1997).  Gutwin  and  Greenberg 
(Gutwin  and  Greenberg  1999)  define  workspace  awareness  as  the  "up-to-the-moment 
understanding  of  another  person's  interaction  with  the  shared  workspace".  This  is  a 
specialized  kind  of  situation  awareness,  awareness  about  what  is  going  on  in  the  workspace 
environment  (including  team  members)  by  opposition  to  awareness  of  the  exterior  situation. 
Awareness  of  an  environment  is  created  and  sustained  through  the  "perception-action  cycle" 
which  involves  1)  perception  as  a  way  to  gain  knowledge  about  the  environment  and  2)  the 
exploration  of  the  environment  directed  by  that  knowledge  in  order  to  gain  more  knowledge. 

Methods  for  measuring  situation  awareness  are  also  an  important  issue.  An  example  is 
SAGAT  (Situation  Awareness  Global  Assessment  Method)  proposed  by  Endsley  (Endsley 
1987)  which  provides  an  immediate  snapshot  of  the  user's  situation  awareness  and  an  overall 
measure  of  awareness  drawn  from  a  summation  of  the  snapshot  scores. 

Several  authors  have  shown  the  importance  of  mental  models  and  mental  pictures  as  a  way  of 
gaining  and  maintaining  situation  awareness  (Endsley  1995)  (Matthews  et  al.  1999)  (Lipshitz 
and  Shaul  1997)  (Mulgund  et  al.  1997).  These  models  could  be  obtained  thanks  to  a  cognitive 
task  analysis  (  Gordon  and  Gill  1997).  Klein  (Klein  1993)  identifies  four  classes  of  such 
analyses:  questionnaires  and  interviews,  controlled  observation,  critical  incidents  and 
analytical  methods. 

Other  researchers  examine  how  situation  awareness  models  can  provide  principles  to  develop 
decision  support  systems  that  go  beyond  the  traditional  data  fusion  model  in  order  to  take  into 
account  operators'  cognitive  needs  (Chalmers  1997)  (Paradis  et  al.  1997b)  (Paradis  et  al. 
1997a).  We  can  conjecture  that  work  on  situation  awareness  will  have  an  important  influence 
on  the  data  fusion  community  in  the  coming  years.  This  could  lead  researchers  to  reconsider 
the  role  of  data  fusion  in  the  context  of  a  broader  view  of  the  decision  making  process  based 
on  the  ability  of  the  operator  to  gain  and  maintain  situation  awareness. 

In  the  next  section  we  review  the  Endsley's  foundatinal  paper  on  situation  awareness. 
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2.  Summary  of  the  Paper  "Toward  a  Theory  of  Situation  Awareness  in 
Dynamic  Systems"  By  Mica  R.  Endsley,  Human  Factors,  1995,  37(1),  32-64 

Because  this  paper  is  a  very  important  one  for  anyone  who  wants  to  understand  the  cognitive 
mechanisms  underlying  situation  awareness,  this  summary  has  been  done  by  editing  excerpts 
from  Endsley's  original  paper. 

2.1  Introduction 

The  range  of  problems  confronting  human  factors  practitioners  has  continued  to  grow  over  the 
past  50  years.  The  operator’s  situation  awareness  (SA)  will  be  presented  as  a  crucial  construct 
on  which  decision  making  and  performance  in  such  systems  hinge. 

Endsley's  goal  is  to  show  (a)  the  importance  of  SA  in  decision  making  in  dynamic 
environments  and  the  utility  of  using  a  model  of  decision  making  that  takes  SA  into  account, 
and  (b)  a  theory  of  SA  that  expands  on  prior  work  (Endsley  1987;  Endsley  1988).  True  SA 
involves  far  more  than  merely  being  aware  of  numerous  pieces  of  data.  It  also  requires  a  much 
more  advanced  level  of  situation  understanding  and  a  projection  of  future  system  states  in 
light  of  the  operator’s  pertinent  goals. 

SA  can  be  shown  to  be  important  in  a  variety  of  contexts:  aircraft,  air  traffic  control,  large- 
systems  operations,  tactical  and  strategic  systems  and  many  other  everyday  activities  call  for  a 
dynamic  update  of  the  situation  to  function  effectively. 

The  need  for  SA  applies  in  a  wide  variety  of  environments.  Acquiring  and  maintaining  SA 
becomes  increasingly  difficult.  Because  the  state  of  the  environment  is  constantly  changing, 
often  in  complex  ways,  a  major  portion  of  the  operator's  job  becomes  that  of  obtaining  and 
maintaining  good  SA.  In  analyzing  the  decision  making  of  tactical  commanders,  Kaempf, 
Wolf,  and  Miller  (Kaempf  et  al.  1993,  p.  1110)  reported  that  “recognizing  the  situation 
provided  the  challenge  to  the  decision  maker”,  confirming  SA's  criticality. 

Operators  must  do  more  than  simply  perceive  the  state  of  their  environment.  They  must 
understand  the  integrated  meaning  of  what  they  are  perceiving  in  light  of  their  goals. 
Researchers  in  many  areas  have  found  that  expert  decision  makers  will  act  first  to  classify  and 
understand  a  situation,  immediately  proceeding  to  action  selection. 

There  is  evidence  that  an  integrated  picture  of  the  current  situation  may  be  matched  to 
prototypical  situations  in  memory,  each  prototypical  situation  corresponding  to  a  “correct” 
action  or  decision,  experts  use  pattern-matching  mechanisms  to  draw  on  long-term  memory 
structures  that  allow  them  to  quickly  understand  a  given  situation. 

There  is  a  need  to  more  explicitly  incorporate  the  concept  into  human  factors  design  efforts.  A 
theory  of  SA  that  clearly  defines  the  construct  and  its  relation  to  human  decision  making  and 
performance  is  needed  to  fulfill  this  mission. 

2.2  A  model  of  situation  awareness 

The  present  objective  is  to  define  a  common  ground  for  discussion  using  the  information  that 
is  available  in  order  to  provide  a  starting  point  for  future  work  on  SA.  Klein  (Klein  1989) 
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attention  to  appropriate  critical  cues,  expectancies  regarding  future  states  of  the  situation,  and 
the  tie  between  situation  awareness  and  typical  actions. 

A  model 


Figure  1  provides  a  basis  for  discussing  SA  in  terms  of  its  role  in  the  overall  decision-making 
process.  According  to  this  model,  a  person’s  perception  of  the  relevant  elements  in  the 
environment  from  system  displays  or  directly  by  the  senses,  forms  the  basis  for  his  or  her  SA. 


Figure  1 .  Model  of  situation  awareness  in  dynamic  decision 

Several  major  factors  are  shown  to  influence  this  process.  First,  individuals  vary  in  their 
ability  to  acquire  SA.  This  is  hypothesized  to  be  a  function  of  an  individual's  information¬ 
processing  mechanisms,  influenced  by  innate  abilities,  experience,  and  training.  In  addition, 
the  individual  may  possess  certain  preconceptions  and  objectives  that  can  act  to  filter  and 
interpret  the  environment  in  forming  SA. 

SArwill  also  be  a  function  of  the  system  design  in  terms  of  the  degree  to  which  the  system 
provides  the  needed  information  and  the  form  in  which  it  provides  it.  Other  features  of  the 
task  environment,  including  workload,  stress,  and  complexity  may  also  affect  SA. 
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Definitions  and  terminology 

It  is  first  necessary  to  distinguish  the  term  situation  awareness,  as  a  state  of  knowledge, 
from  the  processes  used  to  achieve  that  state.  These  processes,  which  may  vary  widely 
among  individuals  and  contexts,  will  be  referred  to  as  situation  assessment  or  as  the  process 
of  achieving,  acquiring,  or  maintaining  SA. 

SA  as  defined  here  does  not  encompass  all  of  a  person’s  knowledge.  It  refers  to  only  that 
portion  pertaining  to  the  state  of  a  dynamic  environment.  Established  doctrine,  rules, 
procedures,  checklists,  and  the  like  -  though  important  and  relevant  to  the  decision-making 
process  -  are  fairly  static  knowledge  sources  that  fall  outside  the  boundaries  of  the  term. 

In  addition  SA  is  explicitly  recognized  as  a  construct  separate  from  decision  making  and 
performance.  SA,  decision  making,  and  performance  are  different  stages  with  different 
factors  influencing  them  and  with  wholly  different  approaches  for  dealing  with  each  of  them; 
thus  it  is  important  to  treat  these  constructs  separately. 

Attention,  working  memory,  workload,  and  stress  are  all  related  constructs  that  can  affect  SA 
but  that  can  also  be  seen  as  separate  from  it. 

Here  is  a  general  definition  of  SA  (Endsley  1987;  Endsley  1988): 

Situation  awareness  is  the  perception  of  the  elements  in  the  environment  within  a  volume  of 
time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the 
near  future. 


Level  1  SA:  Perception  of  the  Elements  in  the  Environment 

The  first  step  in  achieving  SA  is  to  perceive  the  status,  attributes,  and  dynamics  of 
relevant  elements  in  the  environment. 


Level  2  SA:  Comprehension  of  Current  Situation 

Comprehension  of  the  situation  is  based  on  a  synthesis  of  disjoined  Level  1  elements. 
Level  2  SA  goes  beyond  simply  being  aware  of  the  elements  that  are  present  to 
include  an  understanding  of  the  significance  of  those  elements  in  light  of  pertinent 
operator  goals. 


Level  3  SA:  Projection  of  Future  Status 


The  ability  to  project  the  future  actions  of  the  elements  in  the  environment  -  a  least  in 
the  very  near  term  -  forms  the  third  and  highest  level  SA.  This  is  achieved  through 
knowledge  of  the  status  and  dynamics  of  those  elements  and  comprehension  of  the 
situation  (both  Level  1  and  Level  2  SA). 


SA  therefore,  is  based  on  far  more  than  simply  perceiving  information  about  the  environment. 
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it  with  operator's  goals,  and  providing  projected  future  states  of  the  environment  that  are 
valuable  for  decision  making. 


Elements 

From  a  design  standpoint,  a  clear  understanding  of  SA  in  a  given  environment  rests  on  a  clear 
elucidation  of  the  elements  in  the  definition  -  that  is,  identifying  which  things  the  operator 
needs  to  perceive  and  understand.  These  are  specific  to  individual  systems  and  contexts,  and 
as  such  are  the  one  part  of  S  A  that  cannot  be  described  in  any  valid  way  across  arenas. 


Time 

SA  is  highly  temporal  in  nature.  That  is,  SA  is  not  necessarily  acquired  instantaneously  but  is 
built  up  over  time.  Thus  it  takes  into  account  the  dynamics  of  the  situation  that  are  acquirable 
only  over  time  and  that  are  used  to  project  the  state  of  the  environment  in  the  near  future. 


Space 

SA  is  highly  spatial  in  many  contexts.  Spatial  information  is  highly  useful  for  determining 
exactly  which  aspects  of  the  environment  are  important  for  SA.  An  operator’s  SA  needs  to 
incorporate  information  on  that  subset  of  the  environment  that  is  relevant  to  tasks  and  goals. 
Within  this  boundary,  the  elements  may  be  further  subdivided  into  levels  of  importance  for 
SA  or  may  assume  a  relevance  continuum.  Elements  may  vary  in  their  relevance  across  time, 
although  they  do  not  generally  fall  out  of  consideration  completely. 


Team  SA 

It  is  possible  to  talk  about  SA  in  terms  of  teams  as  well  as  individuals.  In  many  situations 
several  individuals  may  work  together  as  a  team  to  make  decisions  and  carry  out  actions.  In 
this  case  one  can  think  of  the  overall  team's  SA,  whereby  each  member  has  a  specific  set  of 
SA  elements  about  which  he  or  she  is  concerned,  as  determined  by  each  member’s 
responsibilities  within  the  team. 

Some  overlap  between  each  team  member’s  SA  requirements  will  be  present.  It  is  this  subset 
of  information  that  constitutes  much  of  team  coordination.  Higher  levels  of  SA  may  not  be 
directly  presented  on  displays,  but  may  be  communicated  verbally,  or  if  the  team  members 
possess  a  shared  mental  model,  each  team  member  may  achieve  the  same  higher-level  SA 
without  necessitating  extra  verbal  communication. 


Link  to  Decision  Making 

In  addition  to  forming  the  basis  for  decision  making  as  a  major  input,  SA  may  also  impact  the 
process  of  decision  making  itself.  There  is  considerable  evidence  that  a  person’s  manner  of 
characterizing  a  situation  will  determine  the  decision  process  chosen  to  solve  a  problem. 
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Other  evidence  suggests  that  even  the  way  a  given  problem  is  presented  (or  framed)  can 
determine  how  the  problem  is  solved. 


Link  to  Performance 

The  relationship  between  SA  and  performance,  though  not  always  direct,  can  also  be 
predicted.  In  general,  it  is  expected  that  poor  performance  will  occur  when  SA  is  incomplete 
or  inaccurate,  when  the  correct  action  for  the  identified  situation  is  not  known  or  not 
calculated,  or  when  time  or  some  other  factor  limits  a  person's  ability  to  carry  out  correct 
action. 


2.3  Human  Properties  Affecting  and  Underlying  SA 

This  discussion  first  focus  on  characteristics  of  the  individual,  including  relevant  information¬ 
processing  mechanisms  and  constructs  that  play  a  role  in  achieving  SA.  In  combination,  the 
mechanisms  of  short-term  sensory  memory,  perception,  working  memory,  and  long-term 
memory  form  the  basic  structures  on  which  SA  is  based  (Figure  3). 
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Figure  3.  Mechanisms  of  SA. 
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Preattentive  Processing 

According  to  most  research  on  information  processing,  environmental  features  are  initially 
processed  in  parallel  through  preattentive  sensory  stores  in  which  certain  properties  are 
detected  such  as  spatial  proximity,  color,  simple  properties  of  shapes,  or  movement,  providing 
cues  for  further  focalized  attention.  Those  objects  that  are  most  salient,  based  on 
preattentively  registered  characteristics,  will  be  further  processed  using  focalized  attention  to 
achieve  perception.  Cue  salience,  therefore,  will  have  a  large  impact  on  which  portions  of  the 
environment  are  initially  attended  to,  and  these  elements  will  form  the  basis  for  the  first  level 
of  SA. 


Attention 

The  development  of  attention  in  the  perception  process  acts  to  present  certain  constraints  on  a 
person's  ability  to  accurately  perceive  multiple  items  in  parallel  and,  as  such,  is  a  major  limit 
on  SA.  In  complex  and  dynamic  environments,  attention  demands  resulting  from  information 
overload,  complex  decision  making,  and  multiple  tasks  can  quickly  exceed  a  person’s  limited 
attention  capacity.  Operators  of  complex  systems  frequently  employ  a  process  of  information 
sampling  to  circumvent  this  limit. 

Working  memory  also  plays  an  important  role,  allowing  one  to  modify  attention  deployment 
on  the  basis  of  other  information  perceived  or  active  goals. 

People  are  active  participants  in  determining  which  element  of  the  environment  will  become  a 
part  of  their  (level  1)  SA  by  directing  their  attention  based  on  goals  and  objectives  and  on  the 
basis  of  long-term  and  working  memory. 

Because  the  supply  of  attention  is  limited,  more  attention  to  some  elements  (resulting  in 
improved  SA  on  these  elements),  however,  may  mean  a  loss  of  SA  on  other  elements  once  the 
limit  is  reached,  which  can  occur  rather  quickly  in  complex  environments. 

Limitations  of  attention  may  be  circumvented  to  some  degree  through  the  development  of 
automatic  ity. 


Perception 

The  way  in  which  information  is  perceived  is  directed  by  the  contents  of  both  working 
memory  and  long-term  memory.  Advanced  knowledge  of  the  characteristics,  form,  and 
location  of  information,  for  instance,  can  significantly  facilitate  the  perception  of  information. 

One’s  preconceptions  of  expectations  about  information  will  affect  the  speed  and  accuracy  of 
the  perception  of  that  information.  Repeated  experience  in  an  environment  allows  one  to 
develop  expectations  about  future  events. 

Long-term  memory  stores  also  play  a  significant  role  in  classifying  perceived  information  into 
known  categories  or  mental  representations  as  an  almost  immediate  act  in  the  perception 
process. 
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The  classification  of  information  into  understood  representations  forms  level  1  SA  and 
provides  the  basic  building  blocks  for  the  higher  levels  of  SA.  The  cues  used  to  achieve  this 
classifications  are  important  to  SA.  With  higher  levels  of  expertise,  people  appear  to  develop 
critical  cues  in  the  environment  that  allow  them  to  make  very  fine  classifications. 


Working  memory 

Once  perceived,  information  is  stored  in  working  memory.  Most  of  a  person’s  active 
processing  of  information  must  occur  in  working  memory.  New  information  must  be 
combined  with  existing  knowledge  and  a  composite  picture  of  the  situation  developed  (level  2 
SA).  Projections  of  future  status  (level  3  SA)  and  subsequent  decisions  as  to  appropriate 
courses  of  actions  must  occur  in  working  memory  as  well. 


Long-term  memory > 

Long-term  memory  structures  can  be  used  to  circumvent  the  limitations  of  working  memory. 
The  exact  organization  of  knowledge  in  long-term  memory  has  received  diversified 
characterization,  including  episodic  memory,  semantic  networks,  schemata,  and  mental 
models.  This  discussion  will  focus  on  schemata  and  mental  models  that  have  been  discussed 
as  important  for  effective  decision  making  in  a  number  of  environments. 

Schemata  provide  coherent  frameworks  for  understanding  information,  encompassing  highly 
complex  system  components,  states,  and  functioning. 

A  script  -  a  special  type  of  schema  -  provides  sequences  of  appropriate  actions  for  different 
types  of  task  performance.  Ties  between  schemata  and  scripts  can  greatly  facilitate  the 
cognitive  process  because  an  individual  does  not  have  to  actively  decide  on  appropriate 
actions  at  every  turn  but  will  automatically  know  the  actions  to  take  for  a  given  situation 
based  on  its  associated  script. 

A  related  concept  is  the  mental  model.  “Mechanisms  whereby  humans  are  able  to  generate 
descriptions  of  system  purpose  and  form,  explanations  of  system  functioning  and  observed 
system  states,  and  predictions  of  future  states”. 

Mental  models  can  be  described  as  complex  schemata  that  are  used  to  model  the  behavior  of 
systems. 

A  situation  model  (i.e.  SA)  can  be  matched  to  schemata  in  memory  that  depict  prototypical 
situations  or  states  of  the  system  model.  These  prototypical  classifications  may  be  linked  to 
associated  goals  or  scripts  that  dictate  decision  making  and  action  performance.  This  provides 
a  mechanism  for  single-step,  “recognition-primed”  decision  making  described  earlier.  This 
process  is  hypothesized  to  be  a  key  mechanism. 

A  well  developed  mental  model  provides  (a)  knowledge  of  the  relevant  elements  of  the 
system  that  can  be  used  in  directing  attention  and  classifying  information  in  the  perception 
process,  (b)  a  means  of  integrating  the  elements  to  form  an  understanding  of  their  meaning 
(level  2  SA),  and  (c)  a  mechanism  for  projecting  future  states  of  the  system  based  on  its 
current  state  and  an  understanding  of  its  dynamics  (level  3  SA). 
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The  key  to  using  these  models  to  achieve  SA  rests  on  the  ability  of  an  individual  to  recognize 
key  features  in  the  environment  -  critical  cues  —  that  will  map  to  key  features  in  the  model. 

In  cases  in  which  scripts  have  been  developed  for  given  prototypical  situation  conditions,  the 
load  on  working  memory  for  generating  alternative  behaviors  and  selecting  one  among  them 
is  even  further  diminished. 

A  major  advantage  of  this  mechanism  is  that  the  current  situation  needs  not  be  exactly  like 
one  encountered  before.  This  is  a  result  of  categorization  mapping. 

Of  prime  importance  is  the  fact  that  this  process  can  be  almost  instantaneous  because  of  the 
superior  abilities  of  human  pattern-matching  mechanisms.  When  an  individual  has  a  well- 
developed  mental  model  for  the  behavior  of  particular  systems  or  domains,  the  model  will 
provide  (a)  for  the  dynamic  direction  of  attention  to  critical  cues,  (b)  expectations  regarding 
future  states  of  the  environment  (including  what  to  expect  as  well  as  what  to  not  expect)  based 
on  the  projection  mechanisms  of  the  model,  and  (c)  a  direct,  single-step  link  between 
recognized  situation  classifications  and  typical  actions. 

Schemata  and  mental  models  are  developed  as  a  function  of  training  and  experience  in  a  given 
environment.  With  experience,  recurrent  situational  components  will  be  noticed  along  with 
recurrent  associations  and  causal  relationships. 


Default  information 

Holland  et  al.  (Holland  et  al.  1986)  explanation  includes  “Q-morphism”  in  which  default 
information  for  the  system  is  provided  in  a  higher  layer  of  the  model.  These  default  values 
may  be  used  by  individuals  to  predict  the  system  performance  unless  some  specific  exception 
is  triggered,  in  which  case  the  appropriate  transition  function  for  that  more  detailed 
classification  will  be  used. 

This  feature  allows  people  to  operate  effectively  on  the  basis  of  often  limited  information. 

In  addition,  default  values  for  certain  features  of  a  system  can  be  used  if  exact  current  values 
are  not  known. 

This  provision  of  mental  models  allows  experts  to  have  access  to  reasonable  defaults  that 
provide  more  effective  decisions  than  those  of  novices  who  simply  have  missing  information 
(or  poorer  defaults). 


Confidence  level 

Another  important  aspect  of  SA  concerns  a  person’s  confidence  level  regarding  the  SA.  The 
confidence  level  associated  with  information  can  influence  the  decisions  that  are  using  that 
information.  One  could  hypothesize  a  degree  of  uncertainty  associated  with  validity  of 
features  used  to  make  the  mapping  from  the  real  world  to  categories  in  the  model. 
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Automaticity 

A  form  of  automaticity  can  be  acquired.  Automatic  processing  tends  to  be  fast,  autonomous, 
effortless.  In  relation  to  SA,  automaticity  poses  an  important  question,  however.  To  what 
degree  do  people  who  are  functioning  automatically  have  SA  ? 

Logan  (Logan  1988)  provided  a  detailed  discussion  of  automaticity  in  cognitive  processing 
that  he  maintained  occurs  through  direct-access,  single-step  retrieval  of  actions  to  be 
performed  from  memory.  When  processing  in  this  way,  an  individual  appears  to  be  conscious 
of  the  situational  elements  that  triggered  the  automatic  retrieval  of  information  from  memory 
(SA),  but  he  or  she  probably  will  not  be  conscious  of  the  mechanisms  used  in  arriving  at  the 
resultant  action  selection. 

As  expressed  by  Dreyfus  (Dreyfus  1981),  the  individual  knows  the  what  but  not  the  how.  If 
asked  to  explain  why  a  particular  decision  was  made,  an  individual  will  usually  have  to 
construct  some  rationale  using  logical  processes  to  provide  an  explanation  of  the  action  he  or 
she  actually  chose  in  an  automatic,  non-analytic  manner. 

The  state  of  the  situation  itself ,  however,  can  still  be  verbalized  as  it  is  in  awareness. 

In  addition,  the  degree  to  which  automatic  processing  occurs  without  any  attention  or 
awareness  has  been  questioned.  An  example  of  the  possibility  of  decision  making  without 
conscious  SA  is  that  of  a  person  driving  home  from  work  follows  the  same  predetermined 
path,  stops  at  stoplights. 

The  major  implications  of  the  use  of  automatic  processes  are  (a)  good  performance  with 
minimal  attention  allocation,  (b)  significant  difficulty  in  accurately  reporting  on  the  internal 
models  used  for  such  processing  and  possibly  on  reporting  which  key  environment  feature 
were  related,  and  (c)  unreliability  and  inaccuracy  of  reporting  on  processes  after  the  fact. 
Based  on  this  discussion,  automaticity  is  theorized  to  provide  an  important  mechanism  for 
overcoming  human  information-processing  limitations  in  achieving  SA  and  making  decisions 
in  complex,  dynamic  environments. 


Goals 

S A  is  important  as  needed  for  decision  making  regarding  some  system  or  task. 

Goals  form  the  basis  of  most  decision  making  in  dynamic  environments.  More  than  one  goal 
may  be  operating  simultaneously,  and  these  goals  may  sometimes  conflict.  In  most  systems, 
people  are  not  helpless  recipients  of  data  from  the  environment  but  are  active  seekers  of  data 
in  light  of  their  goals.  In  what  Casson  (Casson  1983)  has  termed  a  top-down  decision  process, 
a  person's  goals  and  plans  direct  which  aspects  of  the  environment  are  attended  to  in  the 
development  of  SA. 

Simultaneously  with  this  top-down  process,  bottom-up  processing  will  occur.  Patterns  in  the 
environment  may  be  recognized  that  will  indicate  that  new  plans  are  necessary  to  meet  active 
goals  or  that  different  goals  should  be  activated.  In  this  way  a  person’s  current  goals  and  plans 
may  change  to  be  responsive  to  events  in  the  environment.  The  alternating  of  top-down  and 
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bottom-up  processing  allows  a  person  to  perform  effectively  in  a  dynamic  environment.  This 
process  also  relates  to  the  role  of  mental  models  and  schemata. 

A  person’s  current  goal(s).  selected  as  the  most  important  among  competing  goals,  will  act  to 
direct  the  selection  of  a  mental  model.  The  selected  goals  will  also  determine  the  frame 
(Casson  1983).  or  focus,  on  the  model  that  is  adopted.  Plans  are  then  devised  for  reaching  the 
goal  using  the  projection  capabilities  of  the  model.  When  scripts  are  available  for  executing 
the  selected  plan,  they  will  be  employed.  When  scripts  are  not  available,  actions  will  have  to 
be  devised  to  allow  for  plan  completion.  As  an  ongoing  process,  an  individual  observes  the 
current  state  of  the  environment,  with  his  or  her  attention  directed  to  environmental  features 
by  the  goal-activated  model  and  interpreted  in  light  of  it. 

When  these  expectations  match  what  is  observed,  all  is  well.  When  they  do  not  match,  this 
signals  to  the  individual  that  something  is  amiss  and  indicates  a  need  to  change  goals  or  plans 
because  of  a  shift  in  situation  classes,  a  revision  of  the  model,  or  selection  of  a  new  model. 
This  process  can  also  act  to  change  current  goal  selection  by  altering  the  relative  importance 
of  goals. 

When  goals  are  incompatible,  their  associated  priority  level  for  the  identified  situation  class 
determines  which  shall  be  invoked.  Similarly,  plans  may  be  altered  or  new  plans  selected  if 
the  feedback  provided  indicates  that  the  plan  is  not  achieving  results  in  accordance  with  its 
projections,  or  when  new  goals  require  new  plans.  Through  learning,  these  processes  can  also 
serve  to  create  better  models,  allowing  for  better  projections  in  the  future. 


Summary 

To  summarize  the  key  features  of  SA  in  this  model,  a  person’s  SA  is  restricted  by  limited 
attention  and  working  memory  capacity.  Where  they  have  been  developed,  long-term  memory 
stores,  most  likely  in  the  form  of  schemata  and  mental  models,  can  largely  circumvent  these 
limits  by  providing  for  the  integration  and  comprehension  of  information  and  the  projection 
future  events  (the  higher  levels  of  SA),  even  on  the  basis  of  incomplete  information  and  under 
uncertainty. 

The  use  of  these  models  depends  on  pattern  matching  between  critical  cues  in  the 
environment  and  elements  in  the  model.  Schemata  of  prototypical  situations  may  also  be 
associated  with  scripts  to  produce  single-step  retrieval  of  actions  from  memory.  SA  is  largely 
affected  by  a  person’s  goals  and  expectations  which  will  influence  how  attention  is  directed, 
how  information  is  perceived,  and  how  it  is  interpreted.  This  top-down  processing  will 
operate  in  tandem  with  bottom-up  processing  in  which  salient  cues  will  activate  appropriate 
goals  and  models.  In  addition,  automaticity  may  be  useful  in  overcoming  attention  limits; 
however,  it  may  leave  the  individual  susceptible  to  missing  novel  stimuli  that  can  negatively 
affect  SA. 


A244 


Figure  4.  Relationship  of  goals  and  mental  models  to  situation  awareness. 


2.4  Task  and  System  Factors 

A  number  of  task  and  system  factors  can  also  be  postulated  to  influence  an  individual’s  ability 
to  achieve  SA. 
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System  Design 

In  this  process,  transmission  error,  defined  as  a  loss  of  information,  can  occur  at  each 
transition.  First  of  all,  the  system  may  not  acquire  all  the  needed  information  (ei). 

Of  the  information  acquired  by  the  system,  not  all  of  it  may  be  displayed  of  the  operator  (e2). 
Finally,  of  the  information  displayed  by  the  system  and  that  directly  acquirable  from  the 
environment,  there  may  be  incomplete  or  inaccurate  transmission  to  the  human  operator  (e3 
and  64).  because  perceptual,  attention  and  working  memory  constraints,  as  discussed  earlier. 

The  first  external  issue  influencing  SA,  therefore,  is  the  degree  to  which  the  system  acquire 
the  needed  information  from  the  environment.  The  second  major  issue  involves  the  display 
interface  for  providing  that  information  to  the  operator. 


Interface  design 

The  way  in  which  information  is  presented  via  the  operator  interface  will  largely  influence  SA 
by  determining  how  much  information  can  be  acquired,  how  accurately  it  can  be  acquired, 
and  to  what  degree  it  is  compatible  with  the  operator’s  SA  needs.  Determining  specific  design 
guidelines  for  improving  operator  SA  through  the  interface  is  the  challenge  fueling  many 
current  research  efforts. 

The  degree  to  which  displays  provide  information  that  is  processed  and  integrated  in  terms 
of  level  2  and  3  SA. 

The  degree  to  which  information  is  presented  in  terms  of  the  operator's  major  goals  will 
positively  affect  SA. 

Considering  that  mental  models  and  schemata  are  hypothesized  to  be  key  tools  for 
achieving  the  higher  levels  of  SA  in  complex  systems,  the  critical  cues  used  for  achieving 
these  mechanisms,  need  to  be  determined  and  made  salient  in  the  interface  design.  In 
particular  those  cues  that  will  indicate  the  presence  of  prototypical  situations  will  be  of 
prime  importance. 

Designs  need  to  take  into  consideration  both  top-down  and  bottom-up  processing.  In  this 
light,  environmental  cues  with  highly  salient  features  will  tend  to  capture  attention  away 
from  current  goal-directed  processing. 

A  major  problem  for  SA  occurs  when  attention  is  directed  to  a  subset  of  information  and 
other  important  elements  are  not  attended  to,  either  intentionally  or  unintentionally.  A 
preferred  design  will  provide  global  SA  -  an  overview  of  the  situation  across  the  operator 
goals  -  at  all  times,  while  providing  the  operator  with  detailed  information  related  to  his 
or  her  immediate  goals,  as  required. 

Although  filtering  out  information  on  relevant  SA  elements  is  hypothesized  to  be 
detrimental,  the  problem  of  information  overload  in  many  systems  must  still  be 
considered. 

One  of  the  most  difficult  and  taxing  parts  of  SA  is  the  projection  of  future  states  of  the 
system.  This  is  hypothesized  to  require  a  fairly  well  developed  mental  model. 

The  ability  to  share  attention  between  multiple  tasks  and  sources  of  information  will  be 
very  important  in  any  complex  system.  System  designs  that  support  parallel  processing  of 
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The  value  added  by  the  SA  concept  is  a  means  of  integrating  these  constructs  in  terms  of  the 
operator’s  overall  goals  and  decision  behavior.  As  such,  this  provides  several  advantages  in 
the  design  process: 

The  integrated  focus  of  SA  provides  a  means  of  designing  for  dynamic,  goal-oriented 
behavior  with  its  constant  shifting  goals. 

It  provides  a  means  of  moving  from  a  focus  on  providing  operators  with  data  to  providing 
operators  with  information. 

It  provides  a  means  of  incorporating  into  the  design  a  consideration  of  the  interplay  of 
elements,  whereas  more  attention  to  some  elements  may  come  at  the  expense  of  others. 
This  integrated  level  of  focus  provides  a  means  for  assessing  the  efficiency  of  a  particular 
design  concept  that  an  examination  of  underlying  constructs  (attention,  working  memory, 
etc.)  does  not  provide. 

A  few  major  design  issues,  however,  poses  a  serious  challenge  to  SA  across  numerous 
systems  to  warrant  special  consideration:  stress,  workload,  complexity,  and  automation. 


Stress 

Several  types  of  stress  factors  exist  that  may  act  to  influence  SA,  including  (a)  physical 
stressors  -  noise  ...  and  (b)  social  psychological  stressors.  Stressors  can  affect  SA  in  a 
number  of  different  ways.  The  first  is  that  under  various  forms  of  stress,  people  tend  to 
narrow  their  field  of  attention  to  include  only  a  limited  number  of  central  aspects.  Premature 
closure,  arriving  at  a  decision  without  exploring  all  information  available,  has  also  been  found 
to  be  more  likely  under  stress.  Complex  tasks  with  multiple  input  sources  appear  to  be 
particularly  sensitive  to  the  effects  of  stressors. 

A  second  way  in  which  stress  may  affect  SA  is  through  the  decrements  in  working  memory 
capacity  and  retrieval.  The  degree  to  which  working  memory  decrements  will  affect  SA 
depends  on  the  resources  available  to  the  individual  operator. 

In  many  dynamic  systems,  high  mental  workload  is  a  stressor  of  particular  importance. 


Complexity 

A  major  factor  creating  a  challenge  for  operator  SA  is  the  increasing  complexity  of  many 
systems.  System  complexity  is  hypothesized  to  negatively  affect  both  operator  workload  and 
SA  through  factors  such  as  an  increase  in  the  number  of  the  system  components,  the  degree  of 
interaction  between  these  components,  and  the  dynamics  or  rate  of  change  of  the  components. 


Automation 

A  lack  of  SA  has  been  hypothesized  to  underlie  the  out-of-the-loop  performance  decrement 
that  can  accompany  automation.  Although  some  of  this  problem  may  result  from  a  loss  of 
manual  skills  under  automation,  SA  is  also  a  critical  component.  Operators  who  have  lost  SA 
mav  he  slower  To  detect  nrnblem.s  and  also  will  renuire  extra  time  to  reorient  themselves  to 
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relevant  system  parameters  in  order  to  proceed  with  problem  diagnosis  and  assumption  of 
manual  performance  when  automation  fails. 


2.5  Errors  in  Situation  Awareness 

From  an  operational  point  of  view,  there  is  major  concern  about  situations  in  which  the 
operator  has  poor  SA,  thus  increasing  the  probability  of  the  undesirable  performance  to 
investigate  the  factors  that  can  lead  to  breakdowns  in  the  SA  portion  of  the  decision  making 
process. 

Level  1  S  A.  At  the  very  lowest  level  a  person  may  simply  fail  to  perceive  certain  information 
that  is  important  for  SA  in  the  assigned  task  (incomplete  SA).  In  the  simplest  case,  this  may 
result  from  a  lack  detectability  or  discriminability  of  the  physical  characteristics  of  the  signal 
in  question,  from  some  physical  obstruction  preventing  perception. 

In  many  cases  in  which  SA  is  incomplete,  the  relevant  signals  or  cues  are  readily  discemable 
but  not  properly  perceived  by  the  subject.  Furthermore,  some  people  appear  to  be  better  than 
others  at  dividing  their  attention  across  different  tasks.  This  problem  is  compounded  by  the 
addition  of  stress,  which  can  affect  the  information  input  stage  through  premature  closure, 
changes  in  factors  attended  to,  and  deterioration  of  the  scanning  process. 

Level  2  SA.  SA  errors  are  most  often  the  result  of  an  inability  to  properly  integrate  or 
comprehend  the  meaning  of  perceived  data  in  light  of  operator  goals.  This  misreading  of  cues 
can  occur  for  several  reasons.  A  novice  will  not  have  the  mental  models  necessary  for 
properly  comprehending  and  integrating  all  of  the  incoming  data  or  for  determining  which 
cues  are  actually  relevant  to  established  goals.  In  other  cases,  a  person  may  incorrectly  select 
the  wrong  model  from  memory,  based  on  a  subset  of  situational  cues,  and  use  this  model  to 
interpret  all  perceived  data.  However,  if  the  wrong  mental  model  is  initially  selected,  based 
on  a  subset  of  cues,  a  representational  error  may  occur.  These  errors  can  be  particularly 
troublesome. 

Even  when  a  person  has  selected  the  correct  model  with  which  to  interpret  and  integrate 
environmental  stimuli,  errors  can  occur.  Certain  pieces  of  data  may  be  mismatched  with  the 
model  or  not  matched  at  all,  resulting  in  a  failure  to  recognize  a  prototypical  situation.  In 
addition,  SA  errors  could  occur  from  overlying  on  the  default  values  embedded  in  the  model. 

When  no  model  exists  at  all,  level  2  SA  must  be  developed  in  working  memory.  An  inability 
to  perform  this  integration  in  an  accurate,  timely  manner  -  resulting  from  insufficient 
knowledge  or  working  memory  limitations,  particularly  under  stress  -  can  also  lead  to 
inaccurate  or  incomplete  SA. 

Level  3  SA.  Finally,  level  3  SA  may  be  lacking  or  incorrect.  Even  if  a  situation  is  clearly 
understood,  it  may  be  difficult  to  accurately  project  future  dynamics  without  a  highly 
developed  mental  model. 

General  factors.  A  few  general  underlying  factors  may  also  lead  to  SA  errors  at  all  three 
levels.  People  who  have  trouble  with  distributed  attention  may  be  having  trouble  in 
maintaining  multiple  goals. 
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A  second  major  type  of  error  affecting  SA  relates  to  the  role  of  habitual  schemata  (or 
automatic ity).  While  the  habitual  schema  is  operating,  the  person  either  is  not  receptive  to  the 
non-habitual  cues  or  does  not  generate  the  appropriate  higher-level  SA  from  the  perception  of 
the  cues  because  the  appropriate  schema  is  suppressed. 

Detection  of  SA  errors.  A  real  issue  concerns  how  people  know  when  their  SA  is  in  error. 
Very  often  they  may  be  completely  unaware  of  how  much  they  do  not  know  or  of  the 
inaccuracy  of  their  internal  representation  of  the  situation. 

References 

Adams.  M.,  Tenney,  Y.,  and  Pew,  R.  (1995).  ‘‘Situation  awareness  and  the  cognitive 
management  of  complex  systems.”  Human  Factors,  37(1),  85-104 
Bass,  E.  J.  (1998).  “Towards  an  intelligent  tutoring  system  for  situation  awareness  training  in 
complex,  dynamic  environments.”  The  Fourth  International  Conference  on  Intelligent 
Tutoring  Systems  (ITS  ’98),  Berlin,  Germany,  26-35 
Bass,  E.  J.,  Emst-Fortin,  S.  T.,  and  Duncan,  P.  C.  (1998).  “An  Intelligent  Debriefing  System 
for  Situation  Awareness  Training.”  The  Eleventh  Florida  Artificial  Intelligence 
Research  Symposium  (FLAIRS  98),  Menlo  Park,  CA,  USA,  168-172 
Bass,  E.  J.,  Emst-Fortin,  S.  T.,  Duncan,  P.  C.,  and  Small,  R.  L.  (1997).  “Training  and  aiding 
situation  awareness:  A  context-based  approach.”  Final  Report  under  US  Air  Force 
contract  -F41624-97-C-5027,  Search  Technology,  Inc.,  Norcross,  GA,  USA. 

Casson,  R.  W.  (1983).  “Schema  in  cognitive  anthropology.”  Annual  review  of  anthropology, 
12,429-462 

Chalmers,  B.  A.  (1997).  “Design  Issues  for  a  Decision  Support  System  for  a  Modern  Frigate.” 
The  Second  Symposium  on  Situational  Awareness  in  the  Tactical  Air  Environment, 
Patuxent  River,  Maryland,  USA 

Dreyfus.  S.  E.  (1981).  “Formal  models  vs.  human  situational  understanding:  Inherent 
limitations  on  the  modeling  of  business  expertise.”  ORC  81-3,  University  of 
California.  Operations  Research  Center,  Berkeley,  CA,  USA. 

Endsley,  M.  R.  (1987).  “SAGAT:  A  methodology  for  the  measurement  of  situation 
awareness.”  NOR  DOC  87-83,  Northrop  Corp.,  Hawthorne,  CA,  USA. 

Endsley,  M.  R.  (1988).  “Design  and  Evaluation  for  Situation  Awareness  Enhancement.”  The 
Human  Factors  and  Ergonomics  Society  32nd  Annual  Meeting,  Santa  Monica,  CA, 
USA,  97-101 

Endsley.  M.  R.  (1995).  “Toward  a  Theory  of  Situation  Awareness  in  Dynamic  Systems.” 
Human  Factors ,  37(1),  32-64 

Gamer,  T.  K.,  and  Assenmacher,  T.  J.  (1997).  “Situational  Awareness  Integrated  Product 
Team  Summary.”  The  Second  Symposium  on  Situational  Awareness  in  the  Tactical  Air 
Environment,  Patuxent  River,  Maryland,  USA 

Gordon,  S.  E.,  and  Gill,  R.  T.  (1997).  “Cognitive  Task  Analysis.”  Naturalistic  Decision 
Making,  Z.  C.  E.,  G.  Klein,  and  K.  A.  Inc.,  eds.,  Lawrance  Erlbaum  Associates 
Publishers.  131-140. 

Gutwin,  C.,  and  Greenberg,  S.  (1999).  “A  Framework  of  Awareness  for  Small  Groups  in 
Shared-Workspace  Groupeware.”  99-1,  Department  of  Computer  Science,  University 
of  Saskatchewan.  Canada. 

Holland,  J.  H.,  Holyoak,  K.  F.,  Nisbett,  R.  E.,  and  Thagard,  P.  R.  (1986).  Induction: 
Processes  of  inference,  learning  and  discovery ,  MIT  Press,  Cambridge. 


A219 


Kaempf,  G.  L.,  Wolf,  S.,  and  Miller,  T.  E.  (1993).  “Decision-making  in  the  AEGIS  combat 
information  center.”  The  Human  Factors  and  Ergonomics  Society  37nd  Annual 
Meeting.  Santa  Monica,  CA,  USA,  1 107-1  111 

Klein,  G.  A.  (1989).  “Recognition-primed  decisions.”  Advances  in  Man-Machine  Systems 
Research,  W.  B.  Rouse,  ed.,  JAI,  Greewich,  CT,  USA. 

Klein,  G.  A.  (1993).  “Naturalistic  Decision  Making:  Implication  for  Design.”  ,  Crew  System 
Ergonomics  Information  Analysis  Center. 

Lipshitz,  R.,  and  Shaul,  O.  B.  (1997).  “Schemata  and  Mental  Models  in  Recognition-Primed 
Decision  Making.”  Naturalistic  Decision  Making,  Z.  C.  E.,  G.  Klein,  and  K.  A.  Inc., 
eds.,  Lawrance  Erlbaum  Associates  Publishers,  293-303. 

Logan,  G.  D.  (1988).  “Automaticity,  resources,  and  memory:  Theoretical  controversies  and 
practical  implications.”  Human  Factors,  30,  583-598 

Matthews,  M.  L.,  Webb,  R.  D.  G.,  and  Bryant,  D.  J.  (1999).  “COGNITIVE  TASK 
ANALYSIS  OF  THE  HALIFAX-CLASS  OPERATIONS  ROOM  OFFICER.”  , 
Department  of  National  Defense,  Valcartier,  Quebec,  Canada. 

Mulgund,  S.,  Rinkus,  G.,  and  Friskie,  J.  (1997).  “OLIPSA:  On-Line  Intelligent  Processor  for 
Situation  Assessment.”  The  Second  Symposium  on  Situational  Awareness  in  the 
Tactical  Air  Environment,  Patuxent  River,  Maryland,  USA 

Onken,  R.  (1997).  “The  cockpit  assistant  system  CASSY  as  an  on-board  player  in  the  ATM 
environment.”  The  First  USA/Europe  AirTrqffic  Management  R&D  Seminar,  Saclay, 
France  http://atm-seminar-97.eurocontrol.fr/onken.htm  (Last  visited  27-March-2000). 

Paradis,  S.,  Chalmers,  B.,  Carling,  R.,  Roy,  J.,  and  Bosse,  E.  (1997a).  “Towards  a  generic 
model  for  situation  and  threat  assessment.”  DREV  -  R  -  9622,  Defence  Research 
Establishment,  Valcartier,  Quebec,  Canada. 

Paradis,  S.,  Chalmers,  B.  A.,  and  Bergeron,  P.  (1997b).  “Toward  a  Joint  Human-Machine 
Model  for  Situation  and  Threat  Assessment.”  The  Second  Symposium  on  Situational 
Awareness  in  the  Tactical  Air  Environment,  Patuxent  River,  Maryland,  USA 

Stout,  R.  J.,  and  Salas,  E.  (1997).  “Understanding  and  Training  Team  Situational  Awareness: 
A  knowledge  Elicitation  Approach.”  The  Second  Symposium  on  Situational  Awareness 
in  the  Tactical  Air  Environment,  Patuxent  River,  Maryland,  USA 


DISTRIBUTION  INTERNE 


1  -  Chef  scientifique 

2  -  Bibliotheque  des  documents 
1  -  Dr.  D.  Kettani  (auteur) 

1  -  D.  Gouin  (auteur) 

1  -  Dr.  E.  Bosse 
1  -  J.  Roy 
1  -  M.  Belanger 
1  -  Dr.  A.C.  Boury-Brisset 
1  -  Dr.  Z.  Maamar 
1  -  Dr.  A.  Guitouni 
1  -  S.  Paradis 
1  -  B.  Chalmers 
1  -  G.  Thibault 


DISTRIBUTION  EXTERNE 


1  -  DRDCIM 


